Setting up a new kind of service for UT mappers

  • Two Factor Authentication is now available on BeyondUnreal Forums. To configure it, visit your Profile and look for the "Two Step Verification" option on the left side. We can send codes via email (may be slower) or you can set up any TOTP Authenticator app on your phone (Authy, Google Authenticator, etc) to deliver codes. It is highly recommended that you configure this to keep your account safe.

Varpu

Novice Nali City Reviewer
May 21, 2001
309
0
0
67
Finland
furpile.com
For NaliCity admins and especially for QaPete:
The service I'm describing here is NOT a competitor for NaliCity. I am planning merely to make it a supplement for it. The service contains a lot of things that make it slow on large loads. It also does NOT have ANY kind or reviews and/or bulletin boards or news. It is just another pure-bred service for mappers.


Have You ever met one of the following:

- downloaded a HUGE map just to notice that there is one gigantic texture file that is not actually used by the puny map

- downloaded a cool map because it was played in a server. After installation You noticed that You could no more play any map that uses UT.utx in any server but on the one having this cool map.

- downloaded a map but could not find the JimboJambo.uax anywhere

- uploaded a map to NaliCity and noticed 27 seconds after starting the upload that You forgot to include the citytex.utx

- after a succesful upload You noticed a horrible BSP error in the red base ceiling. It was easy to fix but NaliCity had already spread the map all over the planet. So, You decided to make a new version, CTF-NoBspHole]IX[.unr

- noticed that some idiote had replaced You map (by requesting update) with something too ugly to mention

Just to mention a few....

Well, this service then, here's what it will do for You:

You can send Your map into the service with FTP. In order to do that, You have to register into the system with a functioning e-mail (to retrieve Your password). The system will only accept maps that belong to You or are not already in the system. This is checked by comparing a list of map names created from the files You uploaded with Your login account. This is to prevent Mr. WiseGuy from ruining Your artwork. You have also opportunity to remove/rename/update Your map in the system. You also have the choise to allow/deny public visibility of Your data.

After You uploaded the package, the map, text files, additional packages etc are teared apart. They are inspected and stored into the system individually. During this process the system generates a preview HTML page of Your map with screenshots, briefings, level info etc... This information is updated every time You make changes to Your data.

When You want to download a map, You have the choise to get all items that were originally zipped with it or You can let the system automatically include only the needed components (the data owner can define for example the readme as an essential component). You can also get only the nice texture pack.

On the download screen You have warnings in case some of the essential packages conflict with the majority of similar packages around. You can still install the thing but at least You were warned !!!

So, that's basically it. Just some automatic package analysis, some interesting statistics and good download system.

If You have any suggestions for more features, I'd like to hear about them as soon as possible.
 

Varpu

Novice Nali City Reviewer
May 21, 2001
309
0
0
67
Finland
furpile.com
Wha !
NC3 - Dunno anything about that !

What I'm hoping (really) is that they EVER set it up like this or at least give this kind of processing a dedicated server. After a few days/nights of programming I noticed that even my 433 MhZ DEC AlphaStation (RISC) chokes up with 30 parallel map processes. To convert those megahertzes to Wintel megahertzes, You should use a factor around 6...8.

The purpose of this server is to give the mappers a good place to give their maps out for beta testing and look for resources like audio clips, texture packages, scripts and so on. This is one of the reasons why the map components are teared apart and stored individually. There is another reason as well - redundancy.

I really would be surprised if NC3 has features like checking package compatibility with 5000 existing maps and the UT delivery. That is just too cpu intensive for a busy host like NC.
 

Varpu

Novice Nali City Reviewer
May 21, 2001
309
0
0
67
Finland
furpile.com
Plagiation or common sense ?

Well, IMO requiring identification on a system where You can change data is based on common sense, not on some invention ... propably has been in use over 10000 years ...

Door: "Knock - knock - knock "
Inside: " Who' there ?" (sniffing)....
Door: "The parrot man ... coming after the yelling"
Inside: "Ok, show me some proof !"
Door: "Here, take this" (stows a ragged parrot feather into a slit in the door)


And I really hope I won't be stepping on someone's toes here =O( ....
If I'm doing that, there's nothing wrong in letting me hear about it.
 

Twrecks

Spectacularly Lucky
Mar 6, 2000
2,606
10
36
In Luxury
www.twrecks.info
Hmm... What does this button do???

Just b4 the PU plug was pulled, certain download features were momentarily implimented on files, none of my doing but very snazzy to say the least. Text files were readable prior to download and zips cantonated by their contents, very spiffy work there.
Cross referencing is what databases do best, however, knowing what a unr file needs is entirely another subject. 'Specially texture files and whatnot. Half the maps dL'd to NC don't have screenies so the chances of authors using anything but standard files is very high, yet what SPs were used would be a benny.
Emailing authors about missing files is what we reviewers may not do and reward the oversight with a fat zero. Though I must admit, after reloading UT on several occassions found maps with SP texture sets I didn't have because of the ommission of subsequent updates. Authors should be responsible for their own work and state as much info required to play their map in the readme, unfortunately most take the SPs for granted and have been known to even use umx files from U1.
All in all, what you propose isn't entirely evil Varpu, just alot of work when you get down to it and find 30% (being kind) of maps don't deserve the attention and consiquently your's or anyone else's time.
Nuff said, I'm begining to ramble.
 

Claw

Weird little hermit on dried frog pills
Nov 3, 2001
929
0
0
Visit site
Re: Hmm... What does this button do???

Originally posted by Twrecks
All in all, what you propose isn't entirely evil Varpu, just alot of work when you get down to it and find 30% (being kind) of maps don't deserve the attention and consiquently your's or anyone else's time.

Isn't that why he offerded to provide that service?
 

Varpu

Novice Nali City Reviewer
May 21, 2001
309
0
0
67
Finland
furpile.com
A lot of work - or not ?

Actually this entire thing is fully automatic. I've already been testing it with a load of maps. Also, it should help getting the missing packs as You don't need to download the entire map pack.
There is also another issue with maps which fortunately has not hit us too hard; the naming of the packages. The Unreal engine has a design fault which inhibits usage of certain package files:
- there can not be two package files sharing the same name (case insensitive) without the suffix. So You can't have DOM-PerilButt.unr and DOM-PerilButt.uax in the same system.
- the package file names are restricted by some operating systems and by the class inheritage system in Unreal engine. Therefore You can't have spaces in file names, nor can You have dots or any other characters that clash with class naming rules in Unreal. Yeah, I know there are a bunch of maps having ][ in them but it seems that Epic programmers did some circumventing here. The best bet is to use ONLY the a-z, A-Z, 0-9 and '-'.
- some of the file names and especially those with ][ in them have a funny effect in UNIX systems causing the wildcards to fail. This means that You get "No match" responses by typing 'ls -R *' in some UNIX directory that contains one of these files.
 

Twrecks

Spectacularly Lucky
Mar 6, 2000
2,606
10
36
In Luxury
www.twrecks.info
WELL ALRIGHTY THEN

Sounds like you (Varpu) are able to look inside the unr file without going thru Ued to see what packages are required, that's pretty cool but does still have the failure of version mismatch.

Cross ref'ing those pakages out and matching them up to exsisting files then catagorizing them to their specific origin is way cool (UT install, Service pack#, custom, etc.)
If you really can do all this "automagically" it would be something to share indeed.
 

Varpu

Novice Nali City Reviewer
May 21, 2001
309
0
0
67
Finland
furpile.com
First of all, I have coded some C++ classes that peep inside UT packages - and among other things - fetch the GUID (Game UniversaL ID) from there. While importing the packages into the database, I've seen a LOT of incidents where someone just picked up an Epic package, renamed it and then used it !. Man - talking about redundancy =O).
The system is also capable of detecting identical packages even if they have been renamed - there is about 30 MAPS around having some other suffix than unr =O).
Then we have those older packages, maybe from Unreal 1, whose package version is below 68, meaning there is no GUID easily obtainable. This is good news for users because obviously those packages can be used even if they are slightly modified .... I calculate a CRC128 for them and then use that as GUID to be able to identify identical packages.

So how fast is this ?

I run the thing in my Alpha Station 455 (Miata) and a typical map is being consumed into the system in 10 seconds. And this REALLY involves a lot more than just copying the zip into syste. Actually the delivery is whacked into pieces and each of them is inspected and packed individually for later access. All readme's, jpg's and other non-unreal stuff (the system checks each file for UT content regardless it's name) will go into bin at this time. The original zips can be obtained from about zillion places, so I don't keep those. The hardest part is to write the information into the database as it is extensively indexed. This is to gain extra speed when reading it.

And when You fetch a map ?
Typical map with 2 custom packages (searched automatically) and a final zip size of 3Mb bakes in 0.5 seconds =O) - after that it is the line that is making the resistance.

So, You can put a map without the Citytex.utx into this system and when getting it back, You will have the three existing versions of that file in Your package. Or, if You think You don't need the map, You can get only the three Citytex.utx packages. Oh, maybe the map.

An additional feature is the automatically generated readme file. It is fairly technical but it gives a lot of information about the map to the reader, especially if (s)he is a mapper.

Well...enuf pimping. I include a screenie from the search screen (Searched for all assault maps to give an idea of the paging system):
 

Varpu

Novice Nali City Reviewer
May 21, 2001
309
0
0
67
Finland
furpile.com
A sample readme file

This is a sample readme that was generaterd by the system. I intentionally chose a map that has a package with many versions in it:
==========================================================================
ReadMe.txt automatically generated by UTCMS
this service was provided for You by Esko 'Varpu' Ilola
Generated: Sat Jan 5 11:16:36 2002
==========================================================================
The package contains the following files:
1 ReadMe.txt (this file)
2 CTF-Enemyatthegates.unr
3 EATGAirstrike.u
4 InfAdds-1.u
5 InfAdds-2.u
6 InfAdds-3.u
==========================================================================
NOTE: If You used the automatic package collection mechanism, there
may have been multiple packages sharing the same name. If this
has been the case, the files will be all included and a '-n' will be
appended to the file name where the n is a number starting from 1.
==========================================================================
File : CTF-Enemyatthegates.unr
Size : 2860985
Date : Wed May 16 14:22:04 2001
Type : Game level file
Package version : 69
Allow download : Yes
Client optional : No
Only for server : No
GUID : 61538C0729B23B4891FC4425D7DEA5E3
Author : DireIce
Title : CTF Enemy At The Gates
Entry text :
Ideal players : 8-18
Has screenshot : Yes
Support for DM : Yes
Support for DOM : No
Support for AS : No
Support for CTF : Yes
Weapons : Enforcer (4), Minigun ][ (6), Sniper rifle (8)
Rocket launcher (6), Flak cannon (4)
Health : Health pack [+100] (1), Health vial [+5] (12), Medic box [+20] (18)
Armor : No
Powerups : Armor ][ (2), Thigh pads (2), UT Invisibility (2)
UT Shield belt (1)
Ammo : Bullet box [10] (10), Flak ammo [10] (6), Minigun ammo [50] (8)
Rocket pack [12] (8)
Bot pathing : Defense point (8), Flag base (2), Path node (194)
Player start (18)
Zoning : Normal zone (1), Sky zone (1)
Lighting : Ordinary light (146), Triggered light (4)
Geometry : Brush (582)
==========================================================================
File : EATGAirstrike.u
Size : 2974
Date : Wed May 16 13:28:00 2001
Type : System or script file
Package version : 69
Allow download : Yes
Client optional : No
Only for server : No
GUID : C0016146863ED511938800A0CCD510DC
==========================================================================
File : InfAdds-1.u
Size : 551398
Date : Sun Mar 26 22:21:20 2000
Type : System or script file
Package version : 69
Allow download : Yes
Client optional : No
Only for server : No
GUID : 807E4ED96403D4119CE00050BAC42A48
==========================================================================
File : InfAdds-2.u
Size : 555131
Date : Wed Jul 12 21:53:42 2000
Type : System or script file
Package version : 69
Allow download : Yes
Client optional : No
Only for server : No
GUID : 826B96AE5418D4119CE00050BAC42A48
==========================================================================
File : InfAdds-3.u
Size : 610710
Date : Sat Nov 25 21:27:56 2000
Type : System or script file
Package version : 69
Allow download : Yes
Client optional : No
Only for server : No
GUID : B7384BB50C6D594F9E6053649ABAA893
==========================================================================