D-talk

Logging best practices

nospam@example.com (Mark van der Velden) — Sun, 04 Mar 2012 12:38:00 +0100

Introduction

This document explains some best practices for application error/debug logging. Application logging is an important "tool" to help identifying problems. If done correctly, and if an infrastructure is in place that allows events to be put into context, it's easier to debug situations and thus makes it easier to solve problems sooner.

While out of scope for this document, I'd like to point out the following tools that can make that possible:

Logstash - http://logstash.net
Graylog2 - http://graylog2.org

Different logging meanings

Logging can mean different things and logging can have different purposes. The logging this document refers to is application and semantic logging.

Application logging (also called "debug" or "error" logging) is intended to keep track of the "low level" application behaviour. Application logging is used to log events, that allows for debugging a problem within an application or to monitor application health. In contrast, a different form of logging could for example be Google Analytics. Tracking (or "logging" actions of) visitors of a website, this type of logging is very different and not something this document refers too.

Why do we log

In order to understand how to log, it must first be clear what we try to accomplish with application logging.

Continue reading "Logging best practices"

A world without cookies

nospam@example.com (Mark van der Velden) — Mon, 27 Jun 2011 22:00:00 +0200

Imagine a world wide web without cookies. And this might not be the strangest thing, since the use of cookies is severely limited since Wednesday the 22nd of June 2011 in the Netherlands. And more countries will follow: http://www.bbc.co.uk/news/technology-12668552.

The Law

The (Dutch) law, that requires a user to agree before storing data, doesn't only apply on HTTP cookies. But in fact any kind of data that is stored on the users computer. Such as; HTML5 storage, flash cookies. But also desktop applications, etc.. The law also states that cookies "required" for certain functionality, are allowed without confirmation. Personally I don't see how anything will change, with this exception in place. And I wonder how many experts were involved into making this law. But that is a subject for another article perhaps...

What are cookies

Cookies are little packages of information stored in the browser of a website visitor, they can contain "small" amounts of data such as an identifiable token or a user preference.

What purpose do cookies serve

Cookies are very generic and can be used for many things, good and bad. The most popular probably being tracking your activity and advertisement. But they are also used to keep a state between requests and to store a preference. Such as "remember me" at a login form, or perhaps "no I do not want to participate in your survey".

Cookie problems

If When the law becomes international, you are potentially violating laws by simply calling something like: setcookie() without the users consent. There are hacks around the typical HTTP cookies, by using flash cookies for example (Oh and btw, here a guide on how to clear those: http://www.macromedia.com/...ngs_manager07.html). And possibly HTML5 storage could play a role here (or any of it's derivatives).

Another thing that has been happening, is visitor awareness and thus browser features. More and more people block cookies to stop advertisement tracking, but unfortunately this also prevents a user to use the features he or she wants to use (such as login sessions, etc.). There is an answer for this and quite a few browser vendor's plan on implementing the "Do Not Track" (http://donottrack.us/) feature, or have already done so. But I'm not too happy with it. The downside of "Do Not Track" is that it's voluntary for website owners and advertisement companies to respect this feature. Other tools include projects such as "Ad Blocker", that only block cookies (and more) for advertisement purposes. It works pretty good, but that is hardly user-friendly.

But, back to "no more cookies"... How do you solve the problem of keeping a state between requests over a stateless protocol?

Some ideas

Well in short, I have some ideas but definitely no real answers. I don't think there is a real answer just yet.

Let's take the example of a login session. Where you want to offer a secure section to your visitors, where they can (e.g.) read their e-mail, privately. A few things come to mind:

Identifying yourself in every relevant request, using headers for example or an argument &userId=42 in the request. This however has some big problems on its own.
Digest authentication on every relevant request.
Take a look at other technologies, such as IPv6 and move authentication to a lower level.
Perhaps use a commit and rollback system. Where you can do a variety of actions and only your password will be asked in the end, either applying or discarding your actions. Requiring you only to enter your login once. Not the most ideal solution for e-mail though, I have to admit. With this system you would re-post your data to next screens in a similar fashion as e.g. a wizard.
Or perhaps we need to move away from complete server-side applications and move to the client, Javascript applications and use the HTTP spec a little better as it was intended and only have web-services.

Many, if not all, of the things I mentioned above would require secure connections (SSL/TLS) to avoid other security problems. Which might not be a bad move anyway.

Personally I think that there is a future, in an improved implementation of digest authentication over SSL. One that uses HMAC and stronger algorithms, SSL would then supply the missing server validation feature. It should also be more strict and not fall back to insecure legacy features.

Conclusion

All in all I firmly believe that the browser should play a big role in this new cookie recipe and should (partially) solve these problems. Also there should be a more clear separation between "generic storage" and authentication versus a simulated persistency. In more perfect world I would vote for a solution that works on other (underlying) layers and make it application agnostic.

I suppose the point I'm trying to make with this article is the following: Take away a feature the entire world uses (since 1996), and wait for the brilliant and creative minds, perhaps such as yourself, to come up with a more innovative feature. It's time for something better!

Another interesting read:

http://www.vsecurity.com/download/papers/WeaningTheWebOffOfSessionCookies.pdf

I made some updates to this article, based on some comments.

PHP Quiz part 4

nospam@example.com (Mark van der Velden) — Tue, 02 Nov 2010 09:18:00 +0100

It has been a while, but here is part 4 of the PHP Quiz series! A few questions to crack your brain about, or perhaps you know them all? Try them and find out! Also do read the idea behind these quizzes, here: The PHP Quiz series

As always, think of the answer before you execute the code or look it up. Codepad might help you run the examples. You can find round three here.

Visibility is key

Now you see me, now you don't

class testClass {
private $fubar = "rabuf";

function test($test) {
var_dump($test->fubar);
}
}

class dummy {
function test($test) {
var_dump($test->fubar);
}
}

$object1 = new testClass;
$object2 = new testClass;
$dummy = new dummy;

$object1->test($object1); // Can $object1 see the private property of object1 ?
$object1->test($object2); // Can $object1 see the private property of object2 ?
$dummy->test($object1); // Can $dummy see the private property of object1 ?

Static, sticky, icky

class test {
public $counter = -1;

public function increment() {
static $cnt = 0;
$this->counter = ++$cnt;
return $this;
}
}

$object1 = new test;
$object1->increment()->increment();

$object2 = new test;

// What will the output be
echo $object2->increment()->counter;

Getting the class

class b {
function getClassA() {
echo get_class($this);
}
function getClassB() {
echo get_class();
}
function getClassC() {
echo __CLASS__;
}
}

class a extends b {
}

$a = new a;

// What will be returned, 'a' or 'b' ?
$a->getClassA();
$a->getClassB();
$a->getClassC();

The strptime function

$result = strptime('2010-11-28', '%Y-%m-%d');

// What is the output?
echo $result['tm_mday'] .'-'. $result['tm_mon'] .'-'. $result['tm_year'];

The oldtimer

for ($i = 0, $loop = 10, $a = 10; $i < $loop; ++$i, $a = 0) {
++$a;
}

// What is the output ?
echo $a;

Forms, buttons and prototypeJS's getInputs()

nospam@example.com (Mark van der Velden) — Tue, 19 Oct 2010 12:36:00 +0200

In forms you use input elements to send your form. But you can also use "button" elements to achieve the same goal. Quite recently someone committed changes into a framework I use often and changed the "input" elements to "button" elements. Normally this is no problem, but in this case it broke some Javascript I had going on.

I used to use the following:

$('myForm').getInputs('submit').each(function (button) {
// ..
});

But with this function you don't get any "button" elements, to solve this situation I changed it to:

$('myForm').select('button', 'input[type=submit]').each(function (button) {
// ..
});

It works fine however simply filtering on 'button' might be too greedy and some tweaking might be required. I use this to to track down which button was pressed by the user and act accordingly (e.g. "Save and close" versus "save").

Drop empty columns with dynamically defined columns and tables

nospam@example.com (Mark van der Velden) — Tue, 17 Aug 2010 13:54:00 +0200

For a migration process I wanted to build in extra validation in some destructive MySQL queries, to eliminate the risk that data might be lost. Of course all data is back-upped, but I rather be safe then sorry.

This is fairly straight forward, create a stored procedure and perform an ALTER statement whenever the previously defined criteria have been met. But I wanted to define a single procedure rather then create one for every table I had to drop columns from. And since you can't simply use variables for column/table names, you have to improvise a little. I came up with the following:

-- Defining the "drop empty column" SP
DELIMITER @@
DROP PROCEDURE IF EXISTS drop_empty_column @@
CREATE PROCEDURE drop_empty_column(
IN itable VARCHAR(50),
IN icolumn VARCHAR(50),
OUT succeeded INT
)
BEGIN

SET @amount = -1;
SET @itable = itable;
SET @icolumn = icolumn;

-- Build the query, with dynamic table and column. Store the result in @amount
SELECT CONCAT('SELECT COUNT(', @icolumn ,') INTO @amount FROM ', @itable ,' WHERE(', @icolumn ,' IS NOT NULL OR ', @icolumn ,' != "");') INTO @testSql;
PREPARE testSqlStmt FROM @testSql;

-- Execute
EXECUTE testSqlStmt;

-- Check if we have 0 rows, else we still have data and we can't drop.
IF @amount = 0 THEN

-- Build the query
SELECT CONCAT('ALTER TABLE ', @itable ,' DROP COLUMN ', @icolumn ,';') INTO @alterSql;

-- Execute
PREPARE alterSqlStmt FROM @alterSql;
EXECUTE alterSqlStmt;

-- Update the status
SELECT 1 INTO succeeded;
ELSE
SELECT 0 INTO succeeded;
END IF;

END@@
DELIMITER ;

-- Conditionally drop the column "deprecated_column" from table "some_table"
CALL drop_empty_column('some_table', 'deprecated_column', @succeeded);

-- Show the status
SELECT @succeeded;

-- Cleanup
DROP PROCEDURE IF EXISTS drop_empty_column;

I'm fairly positive that this could be done in a much better way then that I'm doing here, but this works too. Simply repeat the CALL drop_empty_column(..) for every table/column combination you want to DROP and update the definition of an "empty column" to what you want. Currently it drops the column only if the values contain nothing other then NULL or "" (empty string) values.

Further reading:

What do you get when you mix; MSSQL, PDO and uniqueidentifier?

nospam@example.com (Mark van der Velden) — Tue, 08 Jun 2010 21:58:00 +0200

You get NULL!

Well you get NULL when you don't cast. Say for example you do the following:

$dbh = new PDO([..]);
$stmt = $dbh->prepare('SELECT accountid FROM dbo.Account');
$stmt->execute();

echo $stmt->fetchColumn(); // NULL

But when you do the following:

$stmt = $dbh->prepare('SELECT CAST(accountid AS varchar(36)) accountid FROM dbo.Account');
$stmt->execute();

echo $stmt->fetchColumn(); // "F05C92A1-3119-4206-A123-49A759AC99FB"

I didn't think the casting would be necessary, since according to the manual: http://msdn.microsoft.com/en-us/library/aa226054(SQL.80).aspx the datatype 'uniqueidentifier' has implicit casts with multiple data-types. But I guess it's just one of those things...

Connecting from PHP on a non Microsoft OS to MSSQL with a domain account

nospam@example.com (Mark van der Velden) — Fri, 04 Jun 2010 07:24:00 +0200

I was asked to create a web interface front-end with Microsoft Dynamics CRM as back-end. But I had some troubles setting up the connection, since it has to be done using a domain logon. This doesn't have to be a problem at all, unless your configuration is wrong! In this article I'll explain a few things and point you in the right direction when you have login problems.

As stated earlier, the server running the PHP installation is not Microsoft. In this case a AS400 installation, but it could've been a Linux installation also. I'm using PDO for this article and PHP version 5.2.11. Even if you don't want to use PDO, I recommend using it only for debugging (if possible) since that will give you *most likely* more debug information then the mssql_* family.

When using PDO with a MS-SQL database, you'll need to supply "dblib" as driver and DBLib uses FreeTDS as underlaying library. FreeTDS can be a source of troubles when you're trying to connect, if not configured properly. So I'll kick-off with a little information about it. Don't skip it if you have login problems!

Continue reading "Connecting from PHP on a non Microsoft OS to MSSQL with a domain account"

PHPUnit conditional test based on a PHP version

nospam@example.com (Mark van der Velden) — Mon, 19 Apr 2010 16:47:00 +0200

I had a problem with running test cases on multiple CI environments, where one of the two runs on PHP 5.2 and the other on PHP 5.3. This basically meant that all our pretty PHP 5.3 code caused the builds to fail on the 5.2 only machine. To solve this problem I needed a way to skip tests when the PHP version was less then 5.3.0. Besides the reason I needed this for a -less then ideal- setup. This can also be a generic way to skip certain tests, based on a PHP version.

class someTest extends PHPUnit_Framework_TestCase
{
public function setUp()
{
// Testing if we are dealing with version 5.3.0 or higher
if (!version_compare(PHP_VERSION, '5.3.0', '>=')) {
$this->markTestSkipped('Invalid PHP version, unable to run tests.');
}
}

public function test_testFoo()
{
// .. some awesum test case .. \\
}
}

You can also use the cool @depends annotation of PHPUnit and put the version logic in a test. This has my preference, but it's not always possible. In case you have some code that simply can't be parsed by the older PHP engines.

If you know a better way to do this, please share!

Multiple PHP versions on one webserver

nospam@example.com (Mark van der Velden) — Tue, 03 Nov 2009 08:39:00 +0100

Introduction

This is a blog about running two PHP versions on one webserver and using multiple php.ini files, this combination can be a tricky one to tackle. But luckily one we can tackle quite easily as long as one of the PHP versions is >= 5.2.7. For this example I'll be using Apache, but the webserver flavor doesn't really matter. The most important part is the "PHP_INI_SCAN_DIR" environment variable.

The why

There could be a number of reasons to want what I'm about to talk about. In my case I have a project where I have a legacy code-base, running on a specific PHP version, and a new code-base which will be run on 5.3. Because the new code-base will be a ongoing progress of replacing the old, it first has to run side by side with the legacy code-base. So I wanted my development image to run two PHP versions.

The old code-base used php.ini settings such as a include-path, error reporting, etc. Which will be different from the new code-bas, and those can no-longer be set with the 'php_value' feature of Apaches since the PHP version we'll be using for that runs as (f)CGI rather then as module.

Continue reading "Multiple PHP versions on one webserver"

Javascript printing a popup window

nospam@example.com (Mark van der Velden) — Thu, 29 Oct 2009 15:25:47 +0100

For the impatient, a working example:http://dynom.nl/jquery/print_popup.html

It seems so easy, but I had some trouble printing a popup window containing an image. Whenever I printed the page using the following code it failed.

/**
* FAIL
*/
function printIt() {
var win = window.open('/path/to/image.jpg', 'Image', 'resizable=yes,...');
if (win) {
win.focus();
win.print();
}

return false;
}

So I changed from opening a URL to writing a IMG tag to the opened window, which works like a charm.

/**
* Works like a charm.
*/
function printIt() {
var win = window.open('', 'Image', 'resizable=yes,...');
if (win) {
win.document.writeln('<img src="/path/to/image" alt="image">');
win.document.close();
win.focus();
win.print();
}

return false;
}

And to put it in jQuery terms:

/**
* To put it in jQuery terms:
*/
Popup = {

init : function () {
$('a#action_print').bind('click', Popup.printIt);
},

printIt : function () {
var win = window.open('', 'Image', 'resizable=yes,...');
if (win.document) {
win.document.writeln('<img src="'+ $(this).attr('href') +'" alt="image" />');
win.document.close();
win.focus();
win.print();
}

return false;
}
}

$(document).ready(function () {
Popup.init();
});

Multiple backend session storage handler

nospam@example.com (Mark van der Velden) — Sun, 11 Oct 2009 11:17:00 +0200

Recently I got asked if I knew about a system that supports multiple session back-ends at once. I didn't know about one and since it's not rocket-science I decided to spent a few hours and whoop something up.

For the impatient, checkout: http://github.com/Dynom/SessionHandler

What does it do?

It is a drop-in high-availability storage back-end for PHP sessions by offering a redundant session storage system. It's as easy as including the lib, define the drivers you want to use (e.g. Memcache and MySQL) prepare their configuration/installation and done. If you already have a MySQL server and a Memcache instance running you can set it up in about 5 minutes. It's also easy to extend and write new drivers, just extend the template class and fill in the blanks.

Continue reading "Multiple backend session storage handler"

Sorting of lists in jQuery

nospam@example.com (Mark van der Velden) — Tue, 06 Oct 2009 13:00:00 +0200

A while back I posted a simple sorting algorithm for number sorting, but it doesn't really do the trick for regular strings (e.g. anchors). But since I needed one recently I wrote another small and very simple sorting solution.

For a working example, see: http://dynom.nl/jquery/listsort.php

My.List = {
Sort : function (listSelector) {
var sortedElements, left, right;

// If we have any list items to sort on
if ($(listSelector + ' li').length > 0) {

// Remove the items from the DOM
sortedElements = $(listSelector + ' li').remove();

// Sort, using a custom sorting and compare by anchor contents
sortedElements.sort(function (a, b) {
left = $(a).children('a').html();
right = $(b).children('a').html();

if (left > right) {
return 1;
} else if (right < left) {
return -1;
} else {
return 0;
}
});

// Place the items back on the DOM
$(listSelector).append(sortedElements);
}
}
};

Again, the source should be self explanatory

Click many, call once with jQuery

nospam@example.com (Mark van der Velden) — Tue, 06 Oct 2009 12:28:00 +0200

Recently I wanted to improve they way some anchors where handled in my recent project, it works a bit like adding and removing items to/from a list. But it was possible to trick the interface by quickly clicking multiple times on a link before the AJAX request was finished, while the back-end handled this without any problems, the interface for the visitor couldn't. And since I'm a neat freak and I had a few minutes to spare, I came up with the following.

A working example: http://dynom.nl/jquery/clickmanycallonce.html.

My.items = {
init : function () {
// Bind the real item only once, and the dummy item permanently
// This way the *real* item is only called once, regardless of the amount of clicks
// And the link remains clickable but doesn't trigger the browser to follow it.
$('ul>li>a.item').one('click', My.items.real_action).bind('click', My.items.dummy_action);
},

dummy_action : function () {
$(this).html( $(this).html() + '.' );
return false;
},

real_action : function () {
/* Fancy AJAX call, that shouldn't be executed more then once for every click */
$('div#frame').html($('div#frame').html() + "<br>\nCalling 'real_action' for anchor: " + $(this).html());
return false;
}
};

// When the DOM is ready
$(document).ready(function () {

// Attach the filter to our input and list
My.items.init();

});

While the code is fairly self explanatory and quite simple here the steps in a bit more detail:

Unobtrusively binding, as soon as the DOM is ready.
Binding the real action handler on 'click' action with jQuery's 'one()' event handler
Binding the dummy action handler on 'click' action with the 'bind()' event handler

This situation is probably only favorable if the item you click is removed from the DOM (or has actions re-bound after the call) else it might not be what you want.

The PHP Quiz series

nospam@example.com (Mark van der Velden) — Fri, 25 Sep 2009 07:23:00 +0200

I like solving puzzles, probably one of the reason why I like programming as much as I do. I also like finding challenges and experimenting, as such I came to the idea to start the PHP Quiz series. They contain typical combination of PHP quirks and lesser known features of PHP.

The reason behind the quizzes is not to advocate bad or good coding practices, but it's intended to let you find out the why in all of it. I believe that by understanding what happens it can make you a better programmer and you might spot bugs easier then without knowing what happens.

The series are not ordered in level of difficulty but merely in the order that I found out about them, thought about them or where simply sent in. But in general I think it's safe to say that your knowledge of PHP should be quite a leap forward from novice before you can answer most questions. This however doesn't mean that, once you can do the quizzes flawlessly, that you are a superior programmer. There is a big different between knowing how to design an application and knowing why $array = array(1,1) + array(2,2) only results in an array with two elements.

An overview of all the PHP Quizzes: http://blog.dynom.nl/categories/PHPQuiz_12

A special thanks to the people who sent in code samples, happy quizzing!

-D

PHP Quiz part 3

nospam@example.com (Mark van der Velden) — Mon, 21 Sep 2009 17:35:00 +0200

Welcome to another part of the PHP Quiz series, again some interesting questions to crack your brain about. If you have some nice additions or questions, be sure to leave a comment. Enjoy part three!

As always, think of the answer before you execute the code or look it up. You can find round two here.

Unset cast

What is the type of $a and what is the type of $b

$a = "A";
$b = (unset) $a;
var_dump($a, $b);

Form fun

What will the output be?

Fun with strings

Strings in PHP are versatile, but how versatile are they... What will the output be?

$juggling = "Itffkhmf";
$rox = "Spy";
$b = $c = "";
for ($i=0; $i<strlen($juggling); $i++) {
$b .= $juggling[ $i ];
$b++;
}
for ($i=0; $i<strlen($rox); $i++) {
$c .= $rox[ $i ];
$c--;
}

var_dump( $b, $c );

Continue reading "PHP Quiz part 3"