Building the database

[Fiel]

Yup, Southperry is going to have a database!

However, this database won't be interpreted stuff like sauna.gibbed or mapletip or hidden-street has. This will be, for the most part, raw data. It greatly simplifies the process for me, and people will have to learn how to read the data. It's not too difficult. Also, the data will not be read from some MySQL database, but rather through a simple series of interconnected HTML pages - just like how the data works.

Everything is uploaded for the following:

www.southperry.net/database/Character.html
www.southperry.net/database/Mob.html
www.southperry.net/database/NPC.html
www.southperry.net/database/Map.html

I'll be making a much simpler database soon enough, but this is it for now.

[kleptophobia]

I marry you? No really, this is great. =] Of course I'm going to wait until tomorrow before complaining about the colors =P

I'm going to sleep and hope that that it is fully functional when I wake up, 24 hours of sleep sound good?

[Spadow]

I had a feeling you would do this. Python-style

Finally my eyes won't hurt because of the black background. :)

[Fiel]

I marry you? No really, this is great. =] Of course I'm going to wait until tomorrow before complaining about the colors =P

I'm going to sleep and hope that that it is fully functional when I wake up, 24 hours of sleep sound good?

Well, too bad. The colors are there to stay.

It's a lot harder to change over 5000 HTML pages than to change 1 CSS thing.

[JezzaRules]

Do you need help? I can help you with some stuff (eg. monster info and drops from my perspective)

[Fiel]

Do you need help? I can help you with some stuff (eg. monster info and drops from my perspective)

I'll be uploading the mob stuff and the interpreted JMS drop tables a different day, but thanks. :)

[Russt]

Reminds me of JoeTang's Freewebs site that used to hold the raw skill data files, which I've done a fair amount of sifting through.

Yay for raw data?

[Cheez]

That's alright, but you should have some sort of script to automatically "translate" all the gibberish for people who wouldn't know, or otherwise anything that would add a description of each little thing.

[WandererWillow]

For those curious, this also is not Another World.

[Afrobean]

I can understand doing this in a way that's super easy for you, but wouldn't you be able to extract data to html as you normally do, then parse into a database? Wouldn't it be possible to have a php script that could read the html pages and parse them appropriately as well?

At the very least, wouldn't it be possible for someone else to do this manually? Not that I'm ungrateful or anything, but no matter how good the data is, plain html posted in a highly technical format won't be that impressive to new comers and won't be all that user friendly either.

[SuGaRnSp1C3]

Sweet, thanks Fiel! :f2:

[Chameleonic]

If it stays in its current format I cant see myself using it much. I am a visual person, I need it to be in a more user friendly mode rather than loads of numbers in columns. I need lots of pictures to go with the data.

[Wani]

Woo! A replacement for Sauna! XD Yay! Sauna is horribly outdated, so nice work, Fiel!

Although I gotta agree, having some sort of front-end to it would be nice. :O Or have both available :P

[Corn]

For those curious, this also is not Another World.

-_-. Sheesh....

This is going to be cool. Thanks Fiel!

[Raven]

I'm appreciating this so much right now, but please oh please put the various character things on different pages or at least have those # sublink things (don't know the technical term) on the top, I dread having to scroll down. Also more userfriendly text would be nice, how you would do that I wouldn't know. Also... maybe a single one size thumbnail to the left of the database id number would be friendlier, although whichever path you choose I fully support.

[Insane]

I freakin' love you guys.

I guess it's a good thing I've gotten good at reading pure data from the random data files posted in extractions.
Also, hooray for no eyeburning white background.

But still, wtf @ this not being Another World. I am getting really curious now.

[Fiel]

Everything is uploaded now. I found a faster method to upload. :)

Enjoy guys, I'll be working on a better database soon enough.

[Dyxanije]

Great job guys. Although I'm not used to reading raw data, I'll get used to it :D.

[Blankout]

Is there going to be something like a tutorial that shows us how to understand the data better? I'm pretty sure not a lot of people understand what "move.2.origin.y" is.

[BladeHavoc]

Is there going to be map data? =X Sorry if that's too much to ask.

I love the database, keep it up.

[Yoorah]

Nice! How did I know this was coming? x)

I like the sauna.gibbed.us layout more, though. >.o

[Slip]

Ah... that's nice! 0.58 references on our own site. XD

Hm... I thought so. Only Character, Mob and NPC.

It seems that Fiel's started on Maps (http://www.southperry.net/database/map.html), but haven't gotten around to uploading.

I was hoping for the complete set. xD

[Insane]

He's actually uploading it right now. In the index of the data files, http://www.southperry.net/database/ - you'll see a .7z.filepart file. That be the maps. Right now it's bigger than the others (at 200 mb, whilst the biggest of the others, mobs, is only at 138), so I guess it won't take too much time 'til it's done.

Yeah, I'm bored and investigate stuff like this.

Actually, considering Fiel said this was all for now, mebe I'm looking into this too much and they're just being uploaded for no reason at all.
...
?!
I like how I edited that in just after Fiel posted.

[WandererWillow]

I'm working with Fiel about ways to possibly make this..."prettier".

[Fiel]

He's actually uploading it right now. In the index of the data files, http://www.southperry.net/database/ - you'll see a .7z.filepart file. That be the maps. Right now it's bigger than the others (at 200 mb, whilst the biggest of the others, mobs, is only at 138), so I guess it won't take too much time 'til it's done.

Yeah, I'm bored and investigate stuff like this.

Smart man. The Maps has another hour of uploading left, then I have to extract and chmod.

[Magus]

Oh, that is actually really cool. Good show old chap.

Also, it would be wise to put a link to it somewhere else (such as the main page)

[Fiel]

I've added watermarks to about half of the images in the database. Turns out I've had a few scrapers (http://en.wikipedia.org/wiki/Web_scraping) come by and start mass downloading stuff. I want to at least get credit for the bandwidth and technical know-how used.

[Katie]

Couple of questions Fiel....

First of all are the maps going to be bigger than what shows up? Like if we wanted to use them for sigs, comics, whatnot? And 2nd the other images you have with watermarks, if we see something that we like and want to use it for the above reason's as well, my we request the image without a watermark?

[Hazzy]

Ohsnap, this is epic. >:O

[IsaacGS]

Couple of questions Fiel....

First of all are the maps going to be bigger than what shows up? Like if we wanted to use them for sigs, comics, whatnot? And 2nd the other images you have with watermarks, if we see something that we like and want to use it for the above reason's as well, my we request the image without a watermark?
the only thing that's in the data are minimaps. Full size maps would require a Map Renderer (which isn't a big deal) and a lot of time (Which is a problem). Additionally map renders don't typically have backgrounds so that would still be some trouble.

[Katie]

the only thing that's in the data are minimaps. Full size maps would require a Map Renderer (which isn't a big deal) and a lot of time (Which is a problem). Additionally map renders don't typically have backgrounds so that would still be some trouble.

Oh ok Thanks alot Isaac. Also what about just like objects from maps? Trees, flowers, etc that have the watermark on it?

[Tamekii]

Thats nice Fiel <33
Keep it up @_@

[MechMike]

Is it possible to make the water marks a bit more subtle?


Maybe a DATABASE donation feature that allows you to Download Images without Water Marks?

[Spaz]

How about a CAPTCHA and/or making downloading non-watermarked images members-only?

[Nikkey]

I love Fiel. I love Fiel. Did I say that I love Fiel?

Edit: Seems like people want a more easier-to-understand-database. I should be able to do that, just give me some time to sleep. Preferably 24 hrs.

[Kevin]

Will you put up a database for the Skill.wz file?

[Fiel]

I plan to have a database for all of the WZ files.

[god]

S> better layout for the database :[

[rako]

i can see you are using the dump.py. i can easily get it to extract into a more user friendly format (using css and better site layout). i could even get it into mysql. the site design will be easy then and you can change it easily. so if you need help send me the dump.py file. i'm currently starting as a web designer so i know my stuff. or another option if i dumped it in xml then i could get php to read that so you don't have to worry about the mysql parameters and like mysql you can change site design whenever you like.

i haven't managed to crack the new encryption. converting java to python is just to hard (especially when there are parts missing(lily)) and using raw data i get from demswz (well I'm not very good with archive formats)

[Cyanne]

Is it possible to get monsters from other versions in there, such as time temple monsters, without making copies all the other stuff already in global, like slimes, pigs, and mushrooms?

[Sybaris]

I was looking for the new CWK monsters in the mob.wz database but they arent there yet. Are you going to be updating it to 0.59 somewhen soon?

[Raven]

I was looking for the new CWK monsters in the mob.wz database but they arent there yet. Are you going to be updating it to 0.59 somewhen soon?

Currently they are :)

[Devil]

Hey Fiel,

Uhm I dunno if you're still working on it, but is there any chance you can do the database extraction for all skill tables (skills.wz)?

Especially with the upgraded old 1st~4th job skills lately, this would be nice to actually know some raw skill data, since Nexon themselves haven't altered the descriptions in game... ;)

Tnx in advance m8! :)

[Xakris]

All images are broken. I can't see anything. Is it just me?

[KajitiSouls]

It's not just you Xakris. Either Fiel decided to disable that or the data for the images changed. That's my two guesses anyways...

[Tehidden]

I'm late on this, but very nice. It's nice to see all the raw data.

[Fiel]

It's not just you Xakris. Either Fiel decided to disable that or the data for the images changed. That's my two guesses anyways...

I ran into problems last week during the extraction. I'm currently uploading all the data as I type this.

[Petrified]

Oh veerrryyy nice.

Raw data is (in my case) better then normal, as it gives information you cant normally find.

[ZachAttack]

Definitely a must use resource. Thanks Fiel!

[Petrified]

Will the database eventually have full maps in photo format?

[Fiel]

Will the database eventually have full maps in photo format?

No, due to bandwidth concerns and because I'm not going to bother with full map renders.

[Vormex]

What happened to the links they don't work anymore I use this resource a lot in my server could you put it up again please?

[Kirov]

What happened to the links they don't work anymore I use this resource a lot in my server could you put it up again please?

http://www.southperry.net/forums/showpost.php?p=100690&postcount=1

[Hazzy]

http://www.southperry.net/forums/showpost.php?p=100690&postcount=1

"The database will be restored, but it will only contain data from MSEA."
I still don't see it. :/

[Styles]

its still offline i guess?