Category Archives: Programming

Streaming POST data through PHP cURL Using CURLOPT_READFUNCTION

Well, I haven’t posted here in quite some time… I’m not dead, and don’t plan on completely ditching this blog, but well…

Anyway, onto the article.

I had a PHP application where I wanted to upload part of a large file to some other server.  The naive method may be to simply split the file and upload through cURL, however I wanted to do this without any splitting.  So I needed a way to send a POST request, being able to build the request body on the fly (note, you’ll need to know the total size to be able to send the Content-Length header)

The obvious decision would be to use sockets rather than cURL, but I felt like seeing if it was possible with cURL anyway.  Although I’ll still probably use sockets (because it’s easier in the end), I thought this might (well, not really) be useful to one of the three readers I get every month.

Anyway, if you look at the curl_setopt documentation, you’ll see a CURLOPT_READFUNCTION constant, however, how to really use it doesn’t seem clear (especially with the boundaries for multipart/form-data encoding type).  Also, the documentation is wrong.

Without further ado, here’s some sample code:

<?php

$boundary = '-----------------------------168279961491';
// our request body
$str = "$boundary\r\nContent-Disposition: form-data; name='how_do_i_turn_you'\r\n\r\non\r\n$boundary--\r\n";

// set up cURL
$ch=curl_init('http://example.com/');
curl_setopt_array($ch, array(
 CURLOPT_HEADER => false,
 CURLOPT_RETURNTRANSFER => true,
 CURLOPT_POST => true,
 CURLOPT_HTTPHEADER => array( // we need to send these two headers
 'Content-Type: multipart/form-data; boundary='.$boundary,
 'Content-Length: '.strlen($str)
 ),
 // note, do not set the CURLOPT_POSTFIELDS setting
 CURLOPT_READFUNCTION => 'myfunc'
));

// function to stream data
// I'm not sure what the file pointer $fp does in this context
// but $ch is the cURL resource handle, and $len is how many bytes to read
function myfunc($ch, $fp, $len) {
 static $pos=0; // keep track of position
 global $str;
 // set data
 $data = substr($str, $pos, $len);
 // increment $pos
 $pos += strlen($data);
 // return the data to send in the request
 return $data;
}

// execute request, and show output for lolz
echo curl_exec($ch);
curl_close($ch);

Hopefully the comments give you enough idea how it all works.

Delv into Git

A few days ago, I decided to start a git repository (+ github account) for my XThreads MyBB plugin.  I never really believed this plugin to be complex enough to really require version control, but as I have never used git before, I decided to use it as an opportunity to test it out and gain some experience with it.

I’ve previously had experience with SVN, and felt that it was rather clunky for handling most personal projects, and the idea of a decentralised version control system (DVCS) somewhat appealed to me.  I did some research before diving into git – apparently, one of git’s criticisms is its steep learning curve.  But I still chose it above other systems such as Mercurial, mainly due to its popularity.  I decided to counter this difficult learning curve by using a TortoiseGit to do most of the stuff for me.

So far, I really do quite like this system over SVN:

  • I no longer have to run an SVN server on my computer (don’t really like running background services that much), and/or have to always commit to a remote server
  • Potentially, I could put it on a USB and commit stuff when away from home (except no TortoiseGit; maybe there’s a nice Windows based git GUI somewhere…)
  • It doesn’t have pesky .svn folders all over the place, so you can simply copy out or package the stuff you’re distributing without having to worry about performing an export
  • Being all contained within a folder, you can easily make a backup copy of the repo if you wish, and restore it if you totally trash the repo
  • It seems a lot faster than SVN.  Maybe it’s because I’ve only got a very small amount of code, but it does certainly seem faster, despite the MSYS builds of Git apparently being slower than the Linux ones.

One negative is that it doesn’t have revision numbers like SVN does.  Possibly this is due to it being designed for non-linear development.  Maybe the feature is there somewhere (as git is said to be powerful) or can be implemented in some way, but it appears that the stock system doesn’t support revision numbers (I guess commit dates can be a reasonable proxy for linear development).

Using public key authentication over passwords is an interesting one – I haven’t really thought about it, but my 2 second intuition doesn’t seem to show much benefit.  Maybe it’s because it uses SSH rather than HTTP (SVN)?

As for GitHub, their issue tracker seems to be basic compared to something like Redmine.  I also noticed that it doesn’t seem to have the ability to diff between arbitrary revisions (would be really useful IMO).
But I guess it’s probably sufficient for a personal project (although I would like to be able to diff between multiple revisions).

Overall, I somewhat like this system, and in fact may put a lot of my code under Git version control, if I can get a better USB drive and maybe learn the Git command line (or find a good portable Git GUI).