This page presents some tools to extract your data from a phpBB3 forum that you created somewhere on the Internet.
Mike contacted me in july 2009. He created a forum on ownforum.org and realized this hosting service was very bad. So he wanted to get his data out of there and put it somewhere else.
Except...
PhpBB3 has an administration's console with a "backup" function. I don't know why but people behind phpBB3 decided it was a good idea to allow a disabling of this function.
And at ownforum.org, guess what?
So Mike contacted ownforum.org, to ask for his data. Answer was: we don't do that.
What? I can't get my data? No.
So Mike found me, probably this page, and asked for some help. (He proposed money, like 50 euros. Come on Mike! I do it for free! For fun!) (By the way, 50 euros... let's be serious. This kind of work would cost around 1,000 euros. That's what I would charge if I did it professionally.)
So I helped him. Below is the result.
The programs should work on any Linux system and Windows, maybe others.
You need libcurl and libxml/libxslt.
(curl,
libxml and libxslt)
Under Linux you will need to edit the Makefile to adjust
paths and libraries.
(And maybe the sources, I didn't try to compile it under Linux.)
(It should work as is, but if you have a problem, contact me.)
Here is version 2.0. Some stuff (users' emails) changed at some point of time.
And here the version 1.0 (just in case something is wrong inside 2.0, get 1.0 and try to let the stuff work; I think about the Windows version; some DLLs might be missing or whatever).
Note that this is a big hack, so it is normal if it does not compile or work out of the box. Get in touch with me for some help.
First of all you will need a local PHP/MySQL installation with phpBB3. You will also need to install a local web server, like Apache.
Before getting the data, you must login to the forum (as
administrator) and set the date's format to:
16 Sep 2009, 10:05
It's in the User Control Panel, in
Board preferences.
Both english and greek languages are supported for the date (well, the month).
You will need a lot of space on disk and a lot of time (especially
for posts).
Note that posts creates a local cache, so you stop/restart
the extraction whenever you want. But note that once a page has been
downloaded, it won't be downloaded a second time, even if there
were changes in the meantime. It's a good idea to turn the forum
as "read only" while extracting data (I don't know how to do that,
try by yourself) (maybe it's impossible by the way...).
Run the following steps, in order:
users --url <url> --login <login> --password <password>
users.txt and add the Administrator
user. (For that: duplicate the last line of the file, change the
name to match the administrator of the forum's name and change the first
number to 2.) Your editor must keep the encoding of the
file (if it's UTF8 it must remain UTF8). Use a very basic editor
that treats files as binary data, not doing fancy stuff.
forums --url <url> --login <login> --password <password>
posts --url <url> --login <login> --password <password>
Don't worry about HTML parse errors (like "element a: validity error").
Worry about real warnings (lines starting with WARNING
or warning). Some are not serious, but some are.
To get the warnings in a log file, you can run:
posts --url <url> --login <login> --password <password> > log.txt
Several php files were produced. Copy them all
in the phpBB3 installation you made locally. Then open a browser
and surf the local php files (go to the URL
http://localhost/phpBB3/XXX.php, or wherever you installed
phpBB3) (replace XXX with the correct stuff of course).
Surf the files, in this order:
user_main.php
forums_main.php
topics_main.php
posts_main.php
update_forums_main.php
Several sql files will be produced. You can import
them in your new phpBB3 forum. (Not the local installation,
but the real new place where you want to host the forum.)
The order of import matters. Import in the following order:
users.sql
forums.sql
topics.sql
posts.sql
update_forums.sql
Note that if you have a lot of messages, you may want to
increase the allowed time for php programs and maybe also
the allowed memory. Look for max_execution_time and
memory_limit in php.ini and increase
as you wish.
Note that the sql files might be very big, too
big for MySQL to accept them, especially posts.sql.
Use the utility split to create several smaller
files. (Only tested with posts.sql, may fail with
the others.)
Ah, to import in phpBB, you need to save the files in the
store/ subdirectory of your phpBB installation.
The names cannot be users.sql
and so on, but must be names accepted by the Restore command
of the ACP of phpBB.
Here are names you can use:
backup_1268987483_2504bc6f77038001.sql
backup_1268987503_528fe3d42dc9d29f.sql
backup_1268987523_fe5af3de12444a26.sql
backup_1268987537_833920c8c7831a0f.sql
backup_1268987549_25d0610246d644c1.sql
First name for users.sql, and so on. Same order as above.
In the ACP you can then restore the five files, in the chronological order (from the older to the younger).
Once the sql files have been proceeded into
the new forum, you must reset statistics. Login as
the administrator and in the ACP (Administration Control
Panel) reset and resynchronize everything (maybe not
the board's start date). And purge
the cache, just to be safe.
At this point, you need to add rights to users, or groups of users. And also put users in groups. Otherwise your forums won't be visible. (Learn how to use phpBB3 for that, or ask me in case of problem.) (But really read phpBB documentation, it's all explained. Thanks.)
Verify that everything looks the same. There might be bugs in there, don't trust the result, double and triple check.
Only posts are downloaded, not the images or files or whatever.
Polls are not extracted.
As you can guess, there are many possibilities for the process to fail, or give incorrect results.
Be sure to:
(The list may grow in the future, based on user's input.)
Dates in the new forum may differ from the old one. Maybe setting timezone to UTC in the old forum before getting data removes that problem (not tested).
Well, there are so many weird cases out there I can't list them all, so the best, in case of problem, is to get in touch with me, I surely can help you directly.
Ah, yes. The programs may crash for whatever reason. All this was not coded with a "production level quality" state of mind. This is a hack. Get ready for troubles. The best is to run all this in a virtual machine, based on QEMU or VmWare or whatever, so that you don't destroy your own system.
Don't create a forum if you can't get your data out of it!
I suppose ownforum.org is not the only place where getting your data out is very hard, so the simplest solution is to look for friendly free hosting services. Carefully read the conditions, they may change in time. Be sure you can get your data out at any time!
Hello dead-end reached friends,
my name is Mike and I have created a forum in the free-forum-hosting ownforum.org. Stupid me I haven't checked if I can get the data out of it, and guess what? I couldn't. Having contacted the webmasters at ownforum.org about what is needed to get the database they came back with "We don't share our database". I suggested that I could offer 50 euros for it (ussually that's what other free hosters charge) but they were reluctant. NO, no, no... I imagine adsense by google was getting them a nice lump sum every month.
I have been searching since then in a way to get it by parsing the forum but my programming skills are rusted like iron left in the rain for 10 years... As I was starting to lose hope I bumped by chance to Sed's page describing how he parsed the phpBB forum into his RSS reader. That was the closest I got into and I thought, let's drop him a line to see what he thinks about my issue. Sed like a gentleman, offered to create the parsing algorithm and the scripts to hold the data to restore them in a scratch environment. I knew that it would not be easy so I tried to offer him something as a compensation for his time, he said "NO, I'll do it for the knowledge and the fun". I said fine and invited him over to Greece to make up for the favour...
After quite some thousands of lines of hard coding by Sed and quite some test from my side I had all the vital data and restored them easily into our new home. Sure some loss happened but I didn't care about things easily replaced such as avatars and so on... Sed saved our forum, if it wans't for him we would be hanging by the strings of ownforum puppeteers being there deadlocked without a backup, endangered to lose everything by someone's single click.
Our forums is about remote controlled boats, no brain surgery, but it is our hobby we love it and we have fun, also it is the only Greek forum of it's type so losing it would render us homeless. We are looking up to Sed for all the things he did for us and I strongly reccomend that is you are in a similar situation, Sed is your saviour. Even if the free hoster is providing you the database for a price, screw them and get Sed's free liferaft.
Sed, may always every aspect in your life turn out the way you wish, you are the BEST PROGRAMMER I've ever met, and this comes from a person that has tought programming in the past...
Thanks again for everything!
Mike
Thanks Mike. Oh, by the way, the old forum is here and the new one is here, just for the record. Happy foruming to all!
Contact: sed@free.fr
Created:
Wed, 16 Sep 2009 11:55:11 +0200
Last update:
Mon, 29 Mar 2010 10:29:46 +0200