SQLSTATE[HY000]: General error: 1366 Incorrect string value: ‘\xE2\x80\x8B\xE2\x80\x8B…’ for column

If you get this (or similar) when trying to insert/update in a MySQL database, you table is not able to cope with the extended version of the UTF8 character set.

You will need to convert your table to use the utf8mb4 character set, and utf8mb4_unicode_ci collation.

An example to do so:

ALTER TABLE `your_table_here` CONVERT TO
  CHARACTER SET utf8mb4
  COLLATE utf8mb4_unicode_ci;

Unable to install sass gem on CentOS 6.9 with Ruby 2.4

Something changed recently, preventing a VM from fully provisioning. Tracking it down was a bit of a PiTA.

OS: CentOS 6.9
Ruby: 2.4 – installed from source with the gearlingguy.ruby ansible role
Gems to be installed: sass

$ sudo gem install sass
Fetching: rb-fsevent-0.10.2.gem (100%)
Successfully installed rb-fsevent-0.10.2
Fetching: ffi-1.9.21.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing sass:
    ERROR: Failed to build gem native extension.
    current directory: /usr/local/lib/ruby/gems/2.4.0/gems/ffi-1.9.21/ext/ffi_c
/usr/local/bin/ruby -r ./siteconf20180222-18730-x7wu2n.rb extconf.rb
checking for ffi.h... no
checking for ffi.h in /usr/local/include,/usr/include/ffi... no
checking for shlwapi.h... no
checking for rb_thread_blocking_region()... no
checking for rb_thread_call_with_gvl()... yes
checking for rb_thread_call_without_gvl()... yes
creating extconf.h
creating Makefile
current directory: /usr/local/lib/ruby/gems/2.4.0/gems/ffi-1.9.21/ext/ffi_c
make "DESTDIR=" clean
current directory: /usr/local/lib/ruby/gems/2.4.0/gems/ffi-1.9.21/ext/ffi_c
make "DESTDIR="
Running autoreconf for libffi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
configure.ac:3: error: Autoconf version 2.68 or higher is required
configure.ac:3: the top level
autom4te: /usr/bin/m4 failed with exit status: 63
aclocal: autom4te failed with exit status: 63
autoreconf: aclocal failed with exit status: 63
make: *** ["/usr/local/lib/ruby/gems/2.4.0/gems/ffi-1.9.21/ext/ffi_c/libffi-x86_64-linux"/.libs/libffi_convenience.a] Error 63
make failed, exit code 2
Gem files will remain installed in /usr/local/lib/ruby/gems/2.4.0/gems/ffi-1.9.21 for inspection.
Results logged to /usr/local/lib/ruby/gems/2.4.0/extensions/x86_64-linux/2.4.0/ffi-1.9.21/gem_make.out

The issue is with the ffi ruby gem.

Looking at the releases in github, a recent release updated the required version of autoconf installed on the system. Fine for modern systems, not so much for CentOS 6.9.

The solution is to install the ffi ruby gem with the version prior to this recent change:

$ gem install ffi -v 1.9.18

This then allows sass to be installed without complaint!

Adding extra fields to FOSUserBundle / SonataUserBundle

Sadly, this isn’t really documented (at time of writing).

Adding new protected variables to your User.php entity will not actually create database entries when you try to do a doctrine:schema:update.

While one still needs to have the protected variables in this entity class, along with getters and setters, the actual creation work is within UserBundle/Resources/config/doctrine/User.orm.xml

Here is an example for adding a foreign key:




    

        
            
        

        
            
                
            
        

    

Updating a WordPress database with new domain details

Further to my original quick db update queries, I found the need for a more in-depth approach.

A WordPress site my father works on has weekly backups, but he wanted to see that they actually worked.
To do so, I set up a subdomain vhost on my bytemark server and set about getting the back to work with it.

One of the most annoying/lazy/strange things about WordPress is that it uses serialised arrays within database fields. This makes a simple search and replace fail if you need to do anything more than update the two fields here.

This script works for the database set-up for my father. It may require further tweaks if additional plugins store absolute paths or URLs.

It also could do with a heavy dose of refactoring, but it works for my needs at this point in time 🙂

 'wp3/wp-config.php');
if (!isset($argv[1]))
{
	die("Please tell me where wp-config.php isn");
}

$oldURL = 'http://oldDomainName.com';
$newURL = 'http://backup.of.my.wordpress.site.com';

$oldPath = '/home/chem9598/public_html/';
$newPath = '/var/www/backup_wordpres_site/codebase/htdocs/';

$configFile = $argv[1];
include_once($configFile);

$db = mysql_connect(DB_HOST, DB_USER, DB_PASSWORD);

/*
* 1.1) wp_options - simple options
*/
echo "
1.1) wp_options - simple options: "; $updateOptionsSql = "UPDATE wp_options SET option_value = REPLACE(option_value, '" . $oldURL . "', '" . $newURL . "') WHERE option_name IN ('siteurl', 'home');"; $updateOptionsRes = mysql_query($updateOptionsSql, $db); if (!$updateOptionsRes) echo mysql_error($db) . "
"; else echo mysql_affected_rows ($db) . " rows affected
"; /* * 1.2) wp_options - serialised array options */ echo "
1.2) wp_options - serialised array options: "; $selectOptionsSql = 'SELECT * FROM wp_options WHERE option_value LIKE "%' . $oldURL . '%"'; $selectOptionsRes = mysql_query($selectOptionsSql, $db); if (!$selectOptionsRes) echo mysql_error($db) . "
"; $options = array(); while($selectOptionsRes && $row = mysql_fetch_assoc($selectOptionsRes)) { $options[] = $row; } foreach ($options as $option) { if (substr($option['option_value'], 0, 2) == 'a:') { $optionValue = unserialize($option['option_value']); updateArray($optionValue, $oldURL, $newURL); $option['option_value'] = serialize($optionValue); $updateOptionsSql = "UPDATE wp_options SET option_value = '" . mysql_escape_string($option['option_value']) . "' WHERE option_id = " . $option['option_id']; $updateOptionsRes = mysql_query($updateOptionsSql, $db); if (!$updateOptionsRes) echo mysql_error($db) . "
"; else echo "."; } } echo "
"; /* * 1.3) wp_options - file path */ echo "
1.3) wp_options - file path: "; $selectOptionsSql = 'SELECT * FROM wp_options WHERE option_value LIKE "%' . $oldPath . '%"'; $selectOptionsRes = mysql_query($selectOptionsSql, $db); if (!$selectOptionsRes) echo mysql_error($db) . "
"; $options = array(); while($selectOptionsRes && $row = mysql_fetch_assoc($selectOptionsRes)) { $options[] = $row; } foreach ($options as $option) { if (substr($option['option_value'], 0, 2) == 'a:') { $optionValue = unserialize($option['option_value']); updateArray($optionValue, $oldPath, $newPath); $option['option_value'] = serialize($optionValue); $updateOptionsSql = "UPDATE wp_options SET option_value = '" . mysql_escape_string($option['option_value']) . "' WHERE option_id = " . $option['option_id']; $updateOptionsRes = mysql_query($updateOptionsSql, $db); if (!$updateOptionsRes) echo mysql_error($db) . "
"; else echo "."; } } echo "
"; /* * 2.1) wp_posts - guid */ echo "
2.1) wp_posts - guid: "; $updatePostsSql = "UPDATE wp_posts SET guid = REPLACE(guid, '" . $oldURL . "', '" . $newURL . "');"; $updatePostsRes = mysql_query($updatePostsSql, $db); if (!$updatePostsRes) echo mysql_error($db) . "
"; else echo mysql_affected_rows ($db) . " rows affected
"; /* * 2.2) wp_posts - post_content */ echo "
2.2) wp_posts - post_content: "; $updatePostsSql = "UPDATE wp_posts SET post_content = REPLACE(post_content, '" . $oldURL . "', '" . $newURL . "');"; $updatePostsRes = mysql_query($updatePostsSql, $db); if (!$updatePostsRes) echo mysql_error($db) . "
"; else echo mysql_affected_rows ($db) . " rows affected
"; function debug($var) { echo "<" . "pre>"; print_r($var); echo "<" . "/pre>"; } function updateArray(&$array, $find, $replace) { foreach ($array as $key => &$value) { if (is_array($value)) { updateArray($value, $find, $replace); } else { $array[$key] = str_replace($find, $replace, $value); } } }

Symfony 1.4 Doctrine 1.2 MS SQL Server

Web server: Linux (Ubuntu on my dev setup), Apache, PHP 5.3, Symfony 1.4, Doctrine 1.2.
Database server: Microsoft Windows 2008 Server, MS SQL Server

Trying to get Symfony to talk to the database server has been a painful experience for the last few days. But perseverance has paid off.

Lots of Googling with trial & error has resulted in actually achieving a development setup that will mirror the eventual production setup.

Short version:

  1. Follow the FreeTDS and ODBC setup instructions of http://jamesrossiter.wordpress.com/2011/03/08/connecting-to-microsoft-sql-server-using-odbc-from-ubuntu-server/
  2. Use the following in config/databases.yml
all:
  doctrine:
    class: sfDoctrineDatabase
      param:
        dsn: dblib:dbname=datasourcename;host=sqlserver;
        username: ###
        password: ###

In the above snippit, replace ‘datasourcename’ with whatever you used in /etc/odbc.ini and replace ‘sqlserver’ with the name used in /etc/freetds/freetds.conf & /etc/odbc.ini

It’s late and I’ve been struggling to get this working for some time. I may expand this entry in the future if required.

References and insperation:
1) http://blog.acjacinto.com/2011/11/compiling-php-with-mssql-servers-native.html
2) http://www.microsoft.com/download/en/details.aspx?id=28160
3) http://jamesrossiter.wordpress.com/2011/03/08/connecting-to-microsoft-sql-server-using-odbc-from-ubuntu-server/
4) http://trac.symfony-project.org/wiki/HowToConnectToMSSQLServer

 

Fixing odd characters

One of the biggest complaints that web developers have is their lack of control over end users.

When will we be able to tell them a) not to paste Micosoft Word or b) if you must, paste it into something that doesn’t do formatting 1st.

It has a nasty habit of converting quotes into “smart” quotes as well as messing up characters such as the euro symbol €.

This is the fix I came up with that allows end users to continue to paste from such programs and bring their weird charset issues with them:

function fixChars($text)
{
  $chars = array(
    "/xe2x82xac/" => "&euro;",
    "/xe2x80x99/" => "'",
    "/xe2x80x9c/" => "&#8220;", // open quotes
    "/xe2x80x9d/" => "&#8221;", // close quotes
    "/xe2x80x93/" => "&mdash;",
  
    "/x80/" => "&euro;",
    "/x92/" => "'",
    "/x93/" => "&#8220;", // open quotes
    "/x94/" => "&#8221;", // close quotes
    "/x96/" => "&mdash;",
    "/xa3/" => "&pound;",
  );
  
  $find = array_keys($chars);
  $replace = array_values($chars);
  
  return preg_replace($find, $replace, $text);
}

Downloading SoundCloud Playlists

[Update 21st Aug 2013
Post code replaced with a Github GIST.

Please fork and help improve this script 🙂

 

[Updated 7th Feb 2012
Changes:
+ file names are now as they would be if you downloaded via web browser
+ Playlist is now in reverse date uploaded order]

I like to have music while I’m coding. It breaks the silence while working alone at home.

When I have music playing, it’s best if I don’t have to think about setting up playlists, which is why I used to listen to the radio. The radio has it’s limits though, not enough Ninja Tune I think.

Speaking of Ninja Tune, I <3 their SolidSteel mixes http://soundcloud.com/ninja-tune/sets/solid-steel-radio-shows/

What I don’t like is having to listen via my web browser and their music player in Flash. A recourse hog and not great if you want their music on a device disconnected from the ‘net.

Thankfully, with SoundCloud tracks, you can download any file you wish. The downside to it is that there’s a lot to download. What if you want a whole playlist? A playlist of (currently) 151 tracks for instance?

The script below will download all of the tracks in a set to a directory and create a .m3u playlist at the same time.

Please excuse the justified text and syntax-highlighting gone wrong. View plain, or copy to clipboard and it’ll be fine.

wp-block as a shortcode

In my last post I raved about the excellent plugin wp-blocks by Keir Whitaker and then went on to extend it a little.

Time to extend it a little further, this time by adding a shortcode.

Shortcodes are the handy square bracketed code snippets which conjure up more content and functionality direct from a page or post content.

The wp-blocks plugin is currently intended to work with the theme template files in PHP. Unfortunately (or fortunately?) PHP does not get parsed within the content areas. This does stop malicious code from being included, but it does limit things when you know what you’re doing.

The following code will give you a short code to work with which looks similar to the PHP code snippit you would include in your theme template file.

  /**
   * Gets the block data by 'name' and returns it for use in a content shortcode
   * Requires http://wordpress.org/extend/plugins/wp-blocks/
   *
   * @param string $name
   * @return string
   * @author Iain Cuthbertson
   */
  function get_wp_block_shortcode( $atts ) {
    extract( shortcode_atts( array(
      'name' => '',
    ), $atts ) );

    global $wpdb;
    $block_slug = trim($name);
    $data = (array)$wpdb->get_row("SELECT * FROM {$wpdb->prefix}wpb_content WHERE name = '{$name}' AND active = TRUE");  
    if($data) return get_wrapped_block_content($data);
  }
  
  add_shortcode('get_wp_block', 'get_wp_block_shortcode');

Add that to your theme’s functions.php file and then you will be able to add blocks to your content with, for example:

[get_wp_block name="foo"]