Application Security: Validate uploaded files before processing them

This is not a “how to” but a “what to”.

If you’re a software developer working on a system that allows file uploads, before you allow any code to process the file (i.e.  the file has uploaded, now you want to do something with it), you first need to check the file extension matches the “content” of the actual file.

A simple example: A user may upload a file with an “.ico” file extension, but it’s actually an image file with EXIF data, and your system has an EXIF data vulnerability. You can’t trust the underlying software libraries you use will first validate the file extension matches the actual data embedded in the file.

What this means is the software libraries you use may see the .ico extension, think that it’s OK (just an icon), not run any checks that would ordinarily be run to verify the file on file extensions containing EXIF data, but while processing the file it will still executes EXIF functions and processes the vulnerability.

You think that won’t happen? Think again. That’s an attack vector Black hat hackers use. And it works.

As a software developer, when you think about security you have to assume all the things that are “not likely” to happen are going to be the most likely attack vectors for hackers.

 

Our job as software developers is harder than ever. We have to think every “not likely to happen” scenario is going to be a step in an attack by someone malicious.

PHP: Recursively zip a folder/directory structure

Status: Draft (31 May 2019)

This is a work in progress. I want to record my working as I go.


What I Want

I need to zip a folder structure – ultimately, theme and plug-in folder in WordPress – using PHP.

I’m currently developing and testing on a Windows 10 PC. Ultimately it will need to work in Linux (Ubuntu), but the first step is to it working in Windows.

I started with solutions from https://gist.github.com/MarvinMenzerath/4185113/72db1670454bd707b9d761a9d5e83c54da2052ac and https://stackoverflow.com/questions/1334613/how-to-recursively-zip-a-directory-in-php but the zip file was being created, however, not files were added.

I realised the problem was the relative path for folders/files within the zip. My solution below is a modification of the referenced solutions, where I strip the source path from the fill file path of each folder and file to give the relative path to the source.

So far this works in Windows, next step is to test in Linux. I’m actively working the problem so stay tuned.

 

I created a file “D:\Temp\php\test_zip.php” with the code below.
I execute with the following command (I have XAMPP installed with includes PHP): “D:\Temp\php>D:\xampp\php\php.exe test_zip.php”.

The Code

(Aide: I used https://www.freeformatter.com/html-escape.html to convert raw PHP code to something I can past in between <pre> tags in the Classic WorPress content editor).

<?php
// Make sure the script can handle large folders/files
ini_set('max_execution_time', 600);
ini_set('memory_limit','1024M');

// Start the backup!
Zip('D:\\Temp\\php\\Source\\', './compressed.zip');
Zip('D:\\Temp\\hildas_prod_logs\\', 'D:\\Temp\\php\\backup-test.zip');
testOne('D:\\Temp\\php\\Source\\debug.log', 'D:\\Temp\\php\\compressed_testOne.zip');

echo "Finished" . "\n";


// A quick, reduced test.
// This works in Windows.
function testOne($source, $zipTo)
{
	$zip = new ZipArchive;
	if (file_exists($source) === false)
	{
		echo "Sourece does not exist" . "\n";
		return;
	}
	
	if ($zip->open($zipTo, ZIPARCHIVE::CREATE) === true) 
	{
		$zip->addFile($source, 'source\newname.txt');
		$zip->close();
		echo "OK" . "\n";
	} 
	else 
	{
		echo "FAILED" . "\n";
	}
}


// This works in Windows.
// Source and inspiration: https://gist.github.com/MarvinMenzerath/4185113/72db1670454bd707b9d761a9d5e83c54da2052ac - Marvin Menzerath. (http://menzerath.eu)
// Additional source and inspiration: https://stackoverflow.com/questions/1334613/how-to-recursively-zip-a-directory-in-php
function Zip($source, $destination)
{
    if (!extension_loaded('zip')) 
	{
		echo "Zip extension not loaded\n";
        return false;
    }
	
	if (!file_exists($source)) 
	{
		echo "Source not found:" . $source . "\n";
        return false;
    }

    $zip = new ZipArchive();
    if (!$zip->open($destination, ZIPARCHIVE::CREATE)) 
	{
		echo "Zip not created or opened\n";
        return false;
    }
	
	$raw_source = $source;

    $source = str_replace('\\', '/', realpath($source));
	
	echo "Raw source: " . $raw_source . "\n";
	echo "Clean source: " . $source . "\n";

    if (is_dir($source) === true)
    {
        $files = new RecursiveIteratorIterator(new RecursiveDirectoryIterator($source), RecursiveIteratorIterator::SELF_FIRST);
		
		$sourceWithSeparator = $source . DIRECTORY_SEPARATOR;
        foreach ($files as $file)
        {
            $file = str_replace('\\', '/', $file);

            // Ignore "." and ".." folders
            if( in_array(substr($file, strrpos($file, '/')+1), array('.', '..')) )
                continue;

            $file = realpath($file);

            if (is_dir($file) === true)
            {
				$dir_path = str_replace($sourceWithSeparator, '', $file . DIRECTORY_SEPARATOR);
				
				echo "Directory: " . $file . " (Path: " . $dir_path . ")\n";
                //$zip->addEmptyDir(str_replace($source . '/', '', $file . '/'));
				
				$zip->addEmptyDir($dir_path);
            }
            else if (is_file($file) === true)
            {
				$zip_relative_path = str_replace($sourceWithSeparator, '', $file);
				
				$zip_relative_path = remove_from_start($zip_relative_path, $raw_source);
				
				echo "File: " . $file . " (Path: " . $zip_relative_path . ")\n";
                //$zip->addFromString(str_replace($source . '/', '', $file), file_get_contents($file));
				//$zip->addFile($file, str_replace($source . '/', '', $file));
				$zip->addFile($file, $zip_relative_path);
            }
        }
    }
    else if (is_file($source) === true)
    {
        $zip->addFromString(basename($source), file_get_contents($source));
    }

    return $zip->close();
}



function remove_from_start($full_string, $prefix)
{
	if (substr($full_string, 0, strlen($prefix)) == $prefix) 
	{
		$full_string = substr($full_string, strlen($prefix));
	} 
	
	return $full_string;
}

?>

 

Codeanywhere: Setup a WordPress environment from GitHub repo

Updated: 27 May 2019 (Added “Troubleshooting” at the end).

 

A quick guide to setting up a WordPress development environment in the Codeanywhere online IDE (https://codeanywhere.com/) and linking to an existing GitHub (or other) source control repository (which I assume also contains the WordPress core code).
It’s not entirely obvious so hopefully this helps.

I’m assuming

  • You have already created an account a Codeanywhere.
  • You have already linked Codeanywhere to to your GitHub (or other) source repository.

 

Step 1: In your Codeanywhere Editor page, right-click “Connections” in the left-hand pane and select “New Connection”.

 

Step 2: A “Connection Wizard” dialogue will open. Select “GitHub” (or your relevant source) from the left-side list. A list of repositories will be displayed in the dialogue.

Step 3: Select the repository you want to use. Then un-tick the “Autodetect stack” option below the list (this is what I missed the first time – autodetect doesn’t actually work that well).

 

Step 4: In the next step of the “Connection Wizard” dialogue, scroll down (or search) until you see a “PHP” stack containing “PHP Develppment Stack with Apache, PHP, MySQL, phpMyAdmin and Composer installed”. You can select either Ubuntu or Centos (I work with Ubuntu).
Then select “Create”.

 

That’s it!

A new LAMP environment connected to your GitHub repository should soon available and showing.

Note: I’m new to Codeanywhere (13 May 2019). I’m switching to this IDE from the soon-to-be-closed-and-fucked-by-AWS Cloud9 (c9.io) as my primary IDE for WordPress and so far I’ve only followed the above steps and managed to work [so far successfully] in the SSH terminal. I’m not yet sure about the lifecycle of the Codeanywhere environments yet, or how to run multiple in the editor.

 

Troubleshooting

If you have trouble updating WordPress core or plug-ins within your account and receive a message like this:

To perform the requested action, WordPress needs to access your web server.
Please enter your FTP credentials to proceed.
If you do not remember your credentials, you should contact your web host.

Then add the following line to to wp-config to allow updates:

define( 'FS_METHOD', 'direct' );

See: https://wordpress.stackexchange.com/questions/228591/to-perform-the-requested-action-wordpress-needs-to-access-your-web-server-pleas

Setting up XAMPP for WordPress Development on Windows

Background

I’ve spent the last 3-4 years doing my WordPress development in the Cloud9 IDE (c9.io) until Amazon AWS bought it, fucked up a perfectly good and extremely simple, usable IDe, and now shut down the original service.

I am in the process I switching over to the Codeanywhere IDE as my replacement WordPress development environment, however, I decided it’s time to setup a secondary/backup environment. And being a user of Windows I needed an WAMP (Windows, Apache, MySQL, PHP) style solution to suite.
In the past I’d tried WampServer on my secondary laptop but after leaving and coming back to it, it worked no more. Also, I found it more complicated than I wanted to configure (I hate complicated, particularly in development environments.

So this time I tried something different: XAMPP.

 

System/XAMPP Setup and Configuration

Setup it really easy. Just go to https://www.apachefriends.org/, select the download for you operating system (in my case Windows) and install.

For Windows do the default install, as the following screenshots show.

Step 1:

Step 2:

Step 3:

Step 4:

Step 5:

Step 6:

Step 7:

Step 8:

Step 9 (select your language – English or German):

Step 10: If you ticked “Do you want to start the Control Panel now?” in step 8 then the control panel should start and appear. Services will not start by default (there is an option for that in the control panel Config though).

See the next section for running Apache beside IIS.

 

Conflicts with IIS

If you have IIS installed (as many software developers with Windows will) then you will have conflicts on Port 80. The XAMPP FAQ suggests disabling IIS, but if you’re also a .NET developer like me then you most likely need IIS.

I chose to instead change the port XAMPP uses and this page – How to: Change Your Apache Listen Port Number in XAMPP – Alex Justesen – has all the information you need.

When XAMPP Control Panel first loads you will probably see the following errors:

This is the full log text:

When following the “Alex Justesen” blog post above, the following screenshot show the Apache “Config” and httpd.conf option you need to select.

The general configuration screen for XAMPP (note: I changed my editor from notepad.exe to Notepad++). This is the screen with the “Service and Port Settings” button the “Alex Justesen” post references:

The “Service and Port Settings” screen referenced by the “Alex Justesen” post. I changed “Main Port” to 8080 and “SSL Port” to 4443

Once you finish the the “Alex Justesen” post you can come back to the main XAMPP Control Panel and start Apache and MySQL using the “Start” buttons beside each service:

When starting Apache for the first time you will see the following Windows Security Alert. Select “Allow access” for the service to run:

When starting MySQL for the first time you will see the following Windows Security Alert. Select “Allow access” for the service to run:

You should now see the following success status and logs in XAMPP Control Panel after starting Apache and MySQL:

Select the “Admin” button on the MySQL line to easily open phpmyadmin (you will need this in the section “Setup WordPress”). Note it loads in localhost on your new port:

Select the “Admin” button on the Apache line to easily open XAMPP Dashboard page:

 

XAMPP Control Panel Shortcuts

There are no shortcuts installed to the XAMPP Control Panel (as far as I can tell), so look for xampp-control.exe in your XAMPP installation directory – likely C:\xampp if you use the default install directory.
You will need to run the .exe directly or create your own shortcut to it.

 

Setup Multiple Sites/Ports in XAMPP

Setting up multiple sites in XAMPP (running localhost on multiple ports) was easy.

I followed the following 2 pages:

What you nee to do:

  1. Open “C:\xampp\apache\conf\httpd.conf” (you can do this via the Apache “Config” button in the XAMPP Control Panel.
  2. Look for the line “Listen 80”. Add new Listen lines for any other ports you want to listen for (e.g. I started with 8080 to replace 80, and now I’m adding 8081).
  3. Open “C:\xampp\apache\conf\extra\httpd-vhosts.conf”.
  4. Everything will be commented out, but at the end there is a VirtualHost example like this:
    ##<VirtualHost *:80>
    ##ServerAdmin webmaster@dummy-host2.example.com
    ##DocumentRoot "D:/xampp/htdocs/dummy-host2.example.com"
    ##ServerName dummy-host2.example.com
    ##ErrorLog "logs/dummy-host2.example.com-error.log"
    ##CustomLog "logs/dummy-host2.example.com-access.log" common
    ##</VirtualHost>
  5. Copy and un-comment the entry, then adjust as needed. I created a new site root folder at “C:/xampp/htdocs/8081”, so my entry looks like:
    <VirtualHost *:8081>
    ServerAdmin webmaster@dummy-host2.example.com
    DocumentRoot "D:/xampp/htdocs/8081"
    ServerName dummy-host2.example.com
    ErrorLog "logs/dummy-host2.example.com-error.log"
    CustomLog "logs/dummy-host2.example.com-access.log" common
    </VirtualHost>
  6. “Stop” then “Start” Apache in XAMPP Control panel and you should be good.

 

Setup WordPress

Note: I have core WordPress and all plug-ins under source control.

Steps to get WordPress running:

  1. Login to phpmyadmin for MySQL. You can do this via the “Admin” button beside MySQL in the XAMPP Control Panel.
  2. Create a new database for your WordPress installation.
  3. (I tried creating a new user to user as the DB user in WordPress but couldn’t get it working, so I’m just using root in this environment.
  4. Setup your site files in the website folder. In my environment C:\xampp\htdocs is the root web folder. I created a sub-folder for my WordPress site and cloned my Git repo in there.
  5. Browser to your site. My apache is running under for 8080 and my site is in a folder called “portal”, so I browser to http://localhost:8080/portal.
  6. Go through the usual process for setting up a new WordPress website.
    1. In the screen for database settings, enter the name of the database you created.
    2. Set the username to “root” and leave password blank.
    3. Leave host name as “localhost”.
  7. Your WordPress site should be good to go!

 

The following screenshots show the standard setup screens for a new WordPress site.

Step 5a – Select a language:

Step 5b – Enter your database connection details. I went with the “root” user (no password) that is created when installing MySQL with XAMPP. You will need to first create a database via the phpmyadmin interface (see steps 1 and 2):

Step 5c – WordPress should now be good to setup:

Step 5d – Create your first admin user:

Step 5e – WordPress should be successfully setup:

Step 5e – And now you can login with your new admin user:

 

 

Resolving display of /var/www/html/dportal.com/wp-content/plugins/query-monitor/wp-content/db.php

After getting WordPress running I immediately started to see “/var/www/html/dportal.com/wp-content/plugins/query-monitor/wp-content/db.php” echoed to every page.

A quick search lead me to HTML Output on site migration · Issue #101 · johnbillion/query-monitor and this comment gave me the answer.

I deleted wp-content/db.php file and no more text.

As this stage I’m not sure what is going to happen when I do this for real and start working in both Windows and my Linux-based IDE.

 

Resources

XAMPP FAQ – https://www.apachefriends.org/faq_windows.html

How to Install XAMPP and WordPress Locally on PC/Windows – WPMU DEV

How to Troubleshoot and Fix Common XAMPP and WordPress Issues – WPMU DEV

 

Change a WordPress site URL (when copying or migrating a WordPress site)

This is a post for quick reference. But given 25% (or more) of the Interwebs run on WordPress I want to give it some love too.

Background

Moving (migrating) or copying a WordPress site has some hidden complexity.

While it is easy, there are 3 things you need to do:

  1. Copy the whole website structure, starting with the folder than contains the wp-config.php and other files (it also contains the  “wp-admin”, “wp-content” and “wp-includes” folders).
  2. Change the URL of the site as defined in WordPress (ordinarily set during setup, or within the Settings area).
  3. Update all the saved URLs within the database.

Step 3 is the real gotacha. When you add a link to another page within your site, or insert media, or set a navigation item, it saves the full URL of that item in the database.

For example, if you have a site “https://MySite.com” but want to change it to “https://MySite.com” (or as a software developer, you may want to create a copy of the production site at “https://test.MySite.com”) then after you change the actual URL of the site, all the internal links and images will still be pointing to the original MySite.com.

 

Solution

Many sites have covered how to change a WordPress site URL before, including the official Codex at https://codex.wordpress.org/Changing_The_Site_URL.

I like to keep things simple, so this post is my process:

  1. Backup your site database if you can (I connect directly to the database and run backup scripts, or use phpmyadmin and export).
  2. Update your site URLs by adding the following lines to the top of your active theme’s functions.php file:
    update_option( 'siteurl', 'https://MyNewSite.com' );
    update_option( 'home', 'https://MyNewSite.com' );
    
    1. Browser to your new site (via the IP or URL).
    2. Browse to it again and make sure it’s OK.
    3. Remove the lines from functions.php
    4. Browser to the site again one more time to make sure you can access it now you’ve removed the lines from functions.php.
  3. Upload the interconnect/it “Database Search and Replace Script” script to our site, found at https://interconnectit.com/products/search-and-replace-for-wordpress-databases/.
    1. Install it in a “not easily guessed” sub-directory of the site. This is for security reasons (so other people can’t scan and execute the scripts).
    2. Most fields are filled in. In the “search for…” enter your old URL or IP address, including “http” or “https” (e.g. “https://MySite.com” – you should be using https by now). I don’t end with a trailing forward slash.
      In the “replace with…” field enter the new URL or IP Address (e.g. “https://MyNewSite.com”). Again, not trailing forward slash.
    3. Make sure you do “dry run” to start. This should tell you if you have any problems.
    4. If all is good, do a “live run” and update your database.
    5. At the end, delete the “Database Search and Replace Script” from your site. This is important. If you leave it there bad people may find it and execute it and break your site.

Git: branch commit is in the future

I was just checking some branch commits from one of my developers and noticed the commits where 22 hours in the future.

It turns out they’d changed their system time for testing and forgot to revert it before pushing.

It makes sense when I think about it.

Because git is a distributed system, it would use the local system time when committing.
Git doesn’t track track off a server time.
That means you can be totally disconnected from other systems and still make commits – in that case the only time it knows is the system time.

That doesn’t make it any less confusing or counter-intuitive in an age of servers, synchronisation and centralisation.

Software Development Estimation (and breaking through The Planning Fallacy)

Everyone involved in software development sucks at estimating the time to complete tasks.
Developers. Analysts. Even project managers.

I recently heard a very really simple solution for this problem on a podcast that was discussing The Planning Fallacy.
The solution:

Multiply your estimates by Pi (3.14)

I’ve tried this a few times over the last week and it actually works.
My estimates are finally starting to match reality. And this includes such simple things as writing and responding to emails.

There’s only one problem – that’s jot really a problem – is:

Managers and customers are not going to like hearing the truth

Estimates are going to sound long and expensive, and you will be pushed to reduce them again.

Don’t.
Don’t back down.
Stick to your guns.

The real problem with estimation is not that we suck at it.
The problem is we are forced to cut corners and revise to unrealistic numbers that “sound good” to the customer’s ears.
It never works, and always blow out, and then the shit hits the fan, and stress increases as projects run over time and developers are coerced into working stupid overtime.

If we just do the right thing in the first place and communicate what’s real, everyone knows where they stand and everyone will be happy in the long run.

A Git Strategy

Background

Over the years, I’ve worked on a number of different projects with different Git branching strategies. I’ve also wasted many hours sitting in meeting arguing over what should be a relative non-issue.

The beauty of being the bloke in charge is sometimes you get to skip all the bullshit and just dictate the approach.
Or to be correct in this case, sometimes the approach is dictated for you.

Working remotely with developers who were essentially new to Git and were already facing a lot of learning and change, I found their actions have driven my Git process decisions.
Which is good:

You want to make business and development processes as easy as possible for everyone.

After all, complexity is the enemy of productivity.

I started with GitFlow, which has worked well in the past, but ended up simplifying if further.

 

The Strategy

The Git “Pilot”

One person is designated responsible for reviewing and managing a repository.
All developers should be capable of doing this role – and the role may rotate between developers – but at any point in time one person is the designated “pilot” in control and responsible for the repository.

Workflow

Normal feature development

  • A working branch is created from develop.
  • When work is complete it is merged back into develop via a Pull Request.
  • A release is created.
    • A version number is generated an applied at the start of this process.
    • A new release branch named “release/release_{version}” is created from master.
    • develop is merged into the release branch.
  • The release goes through test environments.
    • Note: Only 1 feature release branch is tested at a time. This reduced management complexity.
    • If a fix is required:
      • A working branch is created from the release branch.
      • The working branch is PR merged back into the release branch.
    • If a production hotfix occurs during this process, develop is merged back into the release branch.
  • A release completes testing and is published to production.
    • Assuming any production hotfixes have already been merged into develop and the release branch.
    • The release branch is merged into master.
    • The release branch is merged into develop (to ensure any fixes are available for continued development).

Hotfixing production issues

  • A working branch is created from master.
  • When work is complete, a release branch is created from master (same versioning and naming process at normal feature development) and the feature branch is merged into the release branch.
  • The release goes through testing.
  • When testing is complete:
    • The release branch is merged back into master.
    • The release branch is merged into develop.
      develop then needs to be merged back into any working branches, and feature release branch currently in test.

Version Numbers

  • Use the format {yyyy}.{mm}.{dd}.{HHmm}
    • {yyyy}: The current 4-digit year at UTC.
    • {mm}: The current 2-digit month at UTC.
    • {dd}: The current 2-digit day at UTC.
    • {HHmm}: The current 24-hour hour and minute at UTC.
    • Example: 2018.02.22.2158
  • Version number is applied as soon as a release is generated.
  • I don’t care about identifying alpha’s, betas, fixes, or otherwise. A release is a release. Use release notes to document details of what is going into a release.

Core Branches

Required:

  • master: The current production code.
  • develop: The current code being developed and tested.
  • release: Each release starts in its own branch.

Optional:

  • feature: Optional node for developer to create working branches under.

Working Branches

Developers can optionally create working branches (known as “feature” branches in GitFlow) under a feature node (e.g. feature/jason_my_work), or directly off the root – at the same level as master, develop, and release.

This “either or” approach came about because my developers were forgetting to create working branches under feature.
In reality this is not a problem in any way and, if anything, made them easier to find and access, so I’m happy for them to stay there.
Good repository maintenance will handle any clutter issues.

Branch Maintenance

After merging working branches back into develop, keep the branch for a “period of time” – generally through testing – we’re happy all code is included and complete.
Periodically delete old working branches.

Release Notes

Each release should contain notes identifying:

  • The type of release – e.g. scheduled feature release, production hotfix, security patching, etc.
  • A summary of changes included in the release.
  • Identify any breaking changes.