Tag Archives: php

Streaming POST data through PHP cURL Using CURLOPT_READFUNCTION

Well, I haven’t posted here in quite some time… I’m not dead, and don’t plan on completely ditching this blog, but well…

Anyway, onto the article.

I had a PHP application where I wanted to upload part of a large file to some other server.  The naive method may be to simply split the file and upload through cURL, however I wanted to do this without any splitting.  So I needed a way to send a POST request, being able to build the request body on the fly (note, you’ll need to know the total size to be able to send the Content-Length header)

The obvious decision would be to use sockets rather than cURL, but I felt like seeing if it was possible with cURL anyway.  Although I’ll still probably use sockets (because it’s easier in the end), I thought this might (well, not really) be useful to one of the three readers I get every month.

Anyway, if you look at the curl_setopt documentation, you’ll see a CURLOPT_READFUNCTION constant, however, how to really use it doesn’t seem clear (especially with the boundaries for multipart/form-data encoding type).  Also, the documentation is wrong.

Without further ado, here’s some sample code:

<?php

$boundary = '-----------------------------168279961491';
// our request body
$str = "$boundary\r\nContent-Disposition: form-data; name='how_do_i_turn_you'\r\n\r\non\r\n$boundary--\r\n";

// set up cURL
$ch=curl_init('http://example.com/');
curl_setopt_array($ch, array(
 CURLOPT_HEADER => false,
 CURLOPT_RETURNTRANSFER => true,
 CURLOPT_POST => true,
 CURLOPT_HTTPHEADER => array( // we need to send these two headers
 'Content-Type: multipart/form-data; boundary='.$boundary,
 'Content-Length: '.strlen($str)
 ),
 // note, do not set the CURLOPT_POSTFIELDS setting
 CURLOPT_READFUNCTION => 'myfunc'
));

// function to stream data
// I'm not sure what the file pointer $fp does in this context
// but $ch is the cURL resource handle, and $len is how many bytes to read
function myfunc($ch, $fp, $len) {
 static $pos=0; // keep track of position
 global $str;
 // set data
 $data = substr($str, $pos, $len);
 // increment $pos
 $pos += strlen($data);
 // return the data to send in the request
 return $data;
}

// execute request, and show output for lolz
echo curl_exec($ch);
curl_close($ch);

Hopefully the comments give you enough idea how it all works.

Safe PHP expressions in templates

In regards to my PHP in Templates MyBB plugin, I’ve been thinking of the possibility of using “safe expressions”, that is, allowing <if> conditionals without allowing admins to enter in “undesirable” PHP.

I came up with an idea yesterday, which I’m giving a shot at.

Basically, there’s probably only two main types of undesirable code:

  • arbitrary modifications of any kind
  • retrieving restricted information

So, any arbitrary PHP code which does neither of the above should be considered “safe”, although I admit that I feel a little edgy over this assumption.

For the first point, there’s only really three ways to perform any modifications in PHP:

  • Assignment operations (=, +=, |=, ++ etc) – this can easily be blocked by finding them in the code (after removing strings); interestingly, PHP doesn’t allow expressions such as $a--$b, instead, they need to be written (IMO properly) as $a-(-$b)
  • Functions/statements (unset, fopen, mysql_query etc) – a whitelist of allowable functions could feasibly block this, although there’d need to be a huge list of allowable functions >_>
  • Executing processes (backtick operator, eg `ls -a`) – just simply block this operator

For the second point, the MyBB template system already allows some information gathering by printing variables (eg $_SERVER[...]) so I won’t consider this to be an issue, instead, I’ll block some constants, such as __FILE__ and PHP_OS, which don’t seem to be easily printable through the MyBB templates system.  The other thing is through functions/statements, which we’re already going to whitelist, so shouldn’t be an issue.

After all that, we just have to consider a few “backdoors”:

  • Executing code within PHP strings, eg "{${phpinfo()}}"
  • Variable function calls, eg $func()

Hopefully, this catches all the bad expressions.

I’m planning on releasing a separate version of the plugin which will not accept these bad expressions.

XThreads v1.2 Released

As mentioned earlier, this plugin was due for an update after I did an update to my Soft Delete plugin.  I got around to finally pushing out an update last night, after pretty much spending a whole day bug testing & fixing, and finding even more bugs – I even managed to stumble on a security issue which I had not yet realised before (not that anyone would’ve probably noticed, as I bet no-one looks through my mess of code), so luckily, I patched that.

Though, as I somewhat expected, I did stuff it up, and this morning, woke up to find that my clever little pre-parser didn’t quite go a well as I thought it did.  I fixed it up, though I did make a bit of a mess with it, having to release 2 versions.  Well, now at v1.22, it seems stable enough, so I guess that’s over and done with.

As I haven’t written about this plugin here before, I’ll write some of my personal thoughts on it.  This is probably my most complex plugin ever (ignoring stuff like my Syntax Highlighter, which is quite algorithmically complex).  Currently at around 5,500 lines of code, this isn’t my biggest (MyPlaza has like >16,000) though I do code rather “compactly” and I’m a bit messy, and chuck stuff on fewer lines than what most other scripters would do.  What makes this so complex, is the amount of integration with MyBB it does, and the amount of core changes it makes to the forum script.  It has to do some rather elaborate hacks in many places to get desired behaviour.  A script like MyPlaza, on the other hand, is much simpler – for one, it mostly adds functionality, rather than change functionality, which is what XThreads tries to do.  However, the result is something I really like.  You can implement a variety of things, from Thread Prefixes to Download Systems to Youtube Video Galleries (many thanks to RateU for the examples) – all without any PHP whatsoever.  Furthermore, as this integrates tightly with MyBB, and in fact, uses many of its features, these systems will often inherit MyBB’s capabilities, such as permissions, ratings and comments, all without some coder having to explicitly implement them.  And that’s not to mention possible integration with other plugins…

Anyway, back to the update, it pretty much adds everything people have asked for, as well as some random ideas that have popped up in my head.  Now XThreads should have a solid base upon which customisations can be built with, as well as handling stuff like very big downloads much better than MyBB does.

So what else is there to do?  I’ve got a long list of stuff that would be nice to implement, but I probably won’t implement many of them unless someone actually wants them.  I think the biggest weakness of this plugin is it being targeted at more advanced users, or perhaps has a bit of a learning curve.  The majority of the MyBB community aren’t terribly knowledgeable, or probably just can’t be bothered with doing things required to make this plugin work.  So maybe something interesting would be a bit of a plugin API to allow third parties (or just the people actually contributing) to make simple modules which add-on functionality without the user having to do a lot of manual edits.  The main thing would be some admin interface – take for example, a gallery – the module would do the necessary edits like in the gallery example, but also have an admin interface for adding gallery components.  Or maybe, I’ll just fall back with a link to edit the gallery forum…  But anyway, something as simple as this will be able to easily leverage the flexibility of XThreads and MyBB to make a powerful system, and perhaps add further specialisation, which has a strong potential to surpass the capability of any other gallery system out there for MyBB.

I’m glad I managed to finalise and push this out this weekend, before I start my second job.  Managing two jobs might be a little taxing on the amount of free time I have, unfortunately.  So this may be the last update in a while.

PHP and Backslash Hell

Backslashes were a fun way to escape certain characters in languages like C (I don’t know if earlier languages have been using the idea, not that I care anyway), for example "\"\\" represents a string containing a double-quote and a backslash character.

Well, it would also make sense to use this for regular expressions (preg_*) functions too.  It’s just that… it gets a little awkward at times.  Once, I wanted to use a regular expression to match a double-backslash (\\) in a string.  For just that, you’d need to put in 8 backslashes in the regular expression – '\\\\\\\\'.  There’s two levels of escaping here – the first one is PHP string parsing (8 -> 4 backslashes) then regular expression parsing (4 -> 2 backslashes).

Nothing much, but, well, dealing with backslashes in regular expressions gets crazy sometimes, especially as most regexs look really confusing anyway.  For example, a bit of code used in my Reverse MyCode Parser, to match captured patterns in regexes:

while(preg_match('~.*(?:^|[^\\\\](?:\\\\\\\\)*)(\(([^?\\\\]|[^?].*?(?:[^\\\\](?:\\\\\\\\)*))\)([.*]\\??|\\?|\\{\d+(?:,\d*)?\\})?)~s', $pattern, $match, PREG_OFFSET_CAPTURE))

Reverse custom MyCode parser

Decided to try writing a basic inverse preg_replace today, knowing that it would be impossible to make a “perfect” algo.

Potentially a number of uses for such, though I’m currently thinking along the lines of a WYSIWYG editor for MyBB (dunno if I’ll make one).  Well, I’ve gone around to making a basic one, which, in fact, probably works on most custom MyCodes posted in the MyBB community forums.  The basic idea is to switch replacement tokens ($1, $2 etc) with the source patterns, and vice versa, which happens to fit nicely with most posted custom MyCodes.

So, in other words, a pattern of \[b\](.*?)\[/b\] and replacement <strong>$1</strong>, after passing through my inverse function, comes out with a pattern of \<strong\>(.*?)\</strong\> and replacement of [b]$1[/b].

It does take into consideration the position and handles repetitions in replacement strings correctly, so a pattern of \[tag\](a)(b)\1\[/tag\] and replacement <strong>$2$1$2$1</strong>, comes out with a pattern of \<strong\>(b)(a)\1\2\</strong\> and replacement of [tag]$2$1$2[/tag].

Obviously, however, it cannot handle all patterns, or even many of them.  The inverse function can’t readily determine what to put in conditions/patterns which aren’t captured or used – it will try to make guesses sometimes, but it’s usually crap.

At the current stage, $0 replacement and nested subpatterns are a little problematic, but probably work.

Continue reading

Soft Delete v1.2

Last night, I updated my Soft Delete plugin for MyBB.  One of the most fiddly plugins I’ve wrote, it mainly includes some fiddly fixes with counters.  Had to add a new deleted thread/post count to fix up pagination displays, which I didn’t think of in earlier versions, so requires a number of changes.  Other main fiddly part was figuring out a way to resolve a conflict with XThreads template in search results – I ended up getting the soft delete plugin to override what XThreads does, so it handles things correctly.

Next plugin update will probably be a v1.2 for XThreads.