Jump to content

[DOWNLOAD] Remove duplicate objects (World JSON Files)


chloe
Message added by AlexRyder

I've merged the old topic with the download topic, and it kinda messed up the original first post, so I had to copy it to the second @chloe's post.

Sorry for that!

Recommended Posts

  • Modz Gold Member

Remove duplicate objects (World JSON Files)

View File

Hello all,

 

I had yesterday a big problem with the World Editor. For whatever reason everything was duplicated. Maybe it was me, maybe a bug, I don't know. Anyway, it was to late to load my last saved work because I changed to much.

 

I was frustrate but than I said to myself: Chloe, calm down, you can code :) So I wrote a little command line tool for the new JSON-World format to remove duplicate objects. Also inside of groups. I didn't had many but if you work on complex structures it can happen that you duplicate a block by accident.

 

The usage is pretty straightforward:

 

ChloeCleanTheWorld.exe "c:\...\awesome.world"

 

The check routine does not considering material. So if you have two objects with the same size and position on top each other with different materials, one of them will be removed!

 

Have fun and take care!

 

Chloe

 

-----------------------------

 

I have changed the code to do deep search of duplicate objects inside groups. Means more duplicate objects are now found :)

 

Please use the tool with care!

 

Chloe

---------------------------------

 

TamaraX reported that she needed to run my tool 5 times before 0 duplicates were found. O don't know were my logic hole is but anyway, I have implemented a multi pass so really every duplicate object should be deleted after a single run:

---------------------------------

Now with UI and installer (and auto update)

---------------------------------

 

 

Did this for Chloe

Tamara

 


 

  • Like 2
  • Thanks 1
Link to comment
Share on other sites

  • Modz Administrators

Thank you for This Chloe.

I have noticed that I seem to get too many copy's of an object and I thought it was just me pressing the Copy button to many times. But perhaps its a bug :/

  • Like 1
Link to comment
Share on other sites

  • Modz Gold Member

Hello all,

 

I had yesterday a big problem with the World Editor. For whatever reason everything was duplicated. Maybe it was me, maybe a bug, I don't know. Anyway, it was to late to load my last saved work because I changed to much. 

I was frustrate but than I said to myself: Chloe, calm down, you can code :) So I wrote a little command line tool for the new JSON-World format to remove duplicate objects. Also inside of groups. I didn't had many but if you work on complex structures it can happen that you duplicate a block by accident.

 

 

The usage is pretty straightforward:

 

ChloeCleanTheWorld.exe "c:\...\awesome.world"

 

 

The check routine does not considering material. So if you have two objects with the same size and position on top each other with different materials, one of them will be removed!

 

If you like to write your own version, here is the (quick and dirty) code:

 

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using Newtonsoft.Json;

namespace World
{
    public interface IXHasChidren
    {
        List<XObject> objects { get; set; }
    }

    public class XRespawn
    {
        public int[] p { get; set; }
        public double r { get; set; }
    }

    public class XObject : IXHasChidren
    {
        private string _tostring = null;

        public string n { get; set; }
        public int[] p { get; set; }
        public int[] r { get; set; }
        public int[] s { get; set; }
        public int[] c { get; set; }
        public string m { get; set; }

        public List<XObject> objects { get; set; }

        public override string ToString()
        {
            if (_tostring != null) return _tostring;

            StringBuilder sb = new StringBuilder();

            sb.Append("{");
            sb.AppendFormat("n:{0}", n);
            sb.Append(",");

            if (n == "group")
            {
                sb.Append("objects:[");
                bool first = true;

                foreach(var obj in objects)
                {
                    if (!first) sb.Append(",");
                    first = false;

                    sb.Append(obj.ToString());
                }
                sb.Append("]");
            }
            else
            {
                sb.AppendFormat("{0}:[{1},{2},{3}]", "p", p[0], p[1], p[2]);

                if (r != null)
                {
                    sb.Append(",");
                    sb.AppendFormat("{0}:[{1},{2},{3}]", "r", r[0], r[1], r[2]);
                }

                if (s != null)
                {
                    sb.Append(",");
                    sb.AppendFormat("{0}:[{1},{2},{3}]", "s", s[0], s[1], s[2]);
                }

                if (c != null)
                {
                    sb.Append(",");
                    sb.AppendFormat("{0}:[{1},{2},{3}]", "c", c[0], c[1], c[2]);
                }
                sb.Append("}");
            }

            sb.Append("}");

            _tostring = sb.ToString();
            return _tostring;
        }
    }

    public class XDoc : IXHasChidren
    {
        public XRespawn respawn { get; set; }
        public double oceanlevel { get; set; }
        public string weather { get; set; }

        public List<XObject>  objects { get; set; }
    }

    class Program
    {
        static void RemoveDuplicates(IXHasChidren container)
        {
            List<XObject> uniqueObjects = new List<XObject>();

            foreach (var objA in container.objects)
            {
                bool duplicate = false;

                foreach (var objB in uniqueObjects)
                {
                    if (objA != objB)
                    {
                        if (objA.ToString() == objB.ToString())
                        {
                            duplicate = true;
                            break;
                        }
                    }

                }

                if (!duplicate) uniqueObjects.Add(objA);
            }

            Console.WriteLine("{0} duplicate objects found.", container.objects.Count - uniqueObjects.Count);
            container.objects = uniqueObjects;
        }

        static void Main(string[] args)
        {
            if (args.Length == 0)
            {
                Console.WriteLine("Please specify a filename :)");
                return;
            }

            string fileName = args[0];
            string json = File.ReadAllText(fileName);
            XDoc doc = JsonConvert.DeserializeObject<XDoc>(json);
            
            RemoveDuplicates(doc);

            foreach (var objA in doc.objects)
            {
                if (objA.n == "group")
                {
                    RemoveDuplicates(objA);
                }
            }

            string cleanJson = JsonConvert.SerializeObject(
                doc,
                Formatting.None,
                new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }
            );

            File.WriteAllText(fileName.Substring(0, fileName.Length - 6) + "_xxx_clean_xxx.world", cleanJson);
        }
    }
}

 

Have fun and take care!

 

Chloe

 

 

ChloeCleanTheWorld.zip

 

 


 

 

I have changed the code to do deep search of duplicate objects inside groups. Means more duplicate objects are now found :)

 

Please use the tool with care! 

 

ChloeCleanTheWorld1_1.zip

 

SOURCE CODE:

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using Newtonsoft.Json;

namespace World
{
    public class XCollectedObject
    {
        public bool isgroup { get; set; }
        public XObject obj  { get; set; }
        public IXHasChildren container  { get; set; }
        public string  hash { get; set; }
    }

    public interface IXHasChildren
    {
        List<XObject> objects { get; set; }
    }

    public class XRespawn
    {
        public int[] p { get; set; }
        public double r { get; set; }
    }

    public class XObject : IXHasChildren
    {
        public string n { get; set; }
        public int[] p { get; set; }
        public int[] r { get; set; }
        public int[] s { get; set; }
        public int[] c { get; set; }
        public string m { get; set; }

        public List<XObject> objects { get; set; }

        public override string ToString()
        {
            StringBuilder sb = new StringBuilder();

            sb.Append("{");
            sb.AppendFormat("n:{0}", n);
            sb.Append(",");

            if (n == "group")
            {
                sb.Append("objects:[");
                bool first = true;

                foreach(var obj in objects)
                {
                    if (!first) sb.Append(",");
                    first = false;

                    sb.Append(obj.ToString());
                }
                sb.Append("]");
            }
            else
            {
                sb.AppendFormat("{0}:[{1},{2},{3}]", "p", p[0], p[1], p[2]);

                if (r != null)
                {
                    sb.Append(",");
                    sb.AppendFormat("{0}:[{1},{2},{3}]", "r", r[0], r[1], r[2]);
                }

                if (s != null)
                {
                    sb.Append(",");
                    sb.AppendFormat("{0}:[{1},{2},{3}]", "s", s[0], s[1], s[2]);
                }

                if (c != null)
                {
                    sb.Append(",");
                    sb.AppendFormat("{0}:[{1},{2},{3}]", "c", c[0], c[1], c[2]);
                }
                sb.Append("}");
            }

            sb.Append("}");

            return sb.ToString();
        }
    }

    public class XDoc : IXHasChildren
    {
        public XRespawn respawn { get; set; }
        public double oceanlevel { get; set; }
        public string weather { get; set; }

        public List<XObject>  objects { get; set; }
    }

    class Program
    {
        static void Collect(ref List<XCollectedObject> list, IXHasChildren container)
        {
            foreach (var obj in container.objects)
            {
                var c = new XCollectedObject()
                {
                    isgroup = obj.n == "group",
                    obj = obj,
                    container = container,
                    hash = obj.ToString()
                };

                list.Add(c);

                if (c.isgroup)
                {
                    Collect(ref list, c.obj);
                }
            }
        }

        static void RemoveDuplicates(ref List<XCollectedObject> list)
        {
            List<XCollectedObject> unique = new List<XCollectedObject>();

            foreach (var objA in list)
            {
                bool duplicate = false;

                foreach (var objB in unique)
                {
                    if (objA.hash == objB.hash)
                    {
                        // remove object from parent container
                        objA.container.objects.Remove(objA.obj);

                        duplicate = true;
                        break;
                    }
                }

                if (!duplicate) unique.Add(objA);
            }

            // find empty containers
            foreach (var objA in list)
            {
                if (objA.isgroup)
                {
                    if (objA.obj.objects.Count == 0)
                    {
                        objA.container.objects.Remove(objA.obj);
                    }
                }
            }
            Console.WriteLine("{0} duplicate objects found.", list.Count - unique.Count);
        }

        static void Main(string[] args)
        {
            if (args.Length == 0)
            {
                Console.WriteLine("Please specify a filename :)");
                return;
            }

            string fileName = args[0];
            string json = File.ReadAllText(fileName);
            XDoc doc = JsonConvert.DeserializeObject<XDoc>(json);

            List<XCollectedObject> allObjects = new List<XCollectedObject>();

            Collect(ref allObjects, doc);
            RemoveDuplicates(ref allObjects);

            string cleanJson = JsonConvert.SerializeObject(
                doc,
                Formatting.None,
                new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }
            );

            File.WriteAllText(fileName.Substring(0, fileName.Length - 6) + "_xxx_clean_xxx.world", cleanJson);
        }
    }
}

 

  • Like 2
Link to comment
Share on other sites

  • Modz Gold Member

TamaraX reported that she needed to run my tool 5 times before 0 duplicates were found. O don't know were my logic hole is but anyway, I have implemented a multi pass so really every duplicate object should be deleted after a single run:

 

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using Newtonsoft.Json;

namespace World
{
    public class XCollectedObject
    {
        public bool isgroup { get; set; }
        public XObject obj  { get; set; }
        public IXHasChildren container  { get; set; }
        public string  hash { get; set; }
    }

    public interface IXHasChildren
    {
        List<XObject> objects { get; set; }
    }

    public class XRespawn
    {
        public int[] p { get; set; }
        public double r { get; set; }
    }

    public class XObject : IXHasChildren
    {
        public string n { get; set; }
        public int[] p { get; set; }
        public int[] r { get; set; }
        public int[] s { get; set; }
        public int[] c { get; set; }
        public string m { get; set; }

        public List<XObject> objects { get; set; }

        public override string ToString()
        {
            StringBuilder sb = new StringBuilder();

            sb.Append("{");
            sb.AppendFormat("n:{0}", n);
            sb.Append(",");

            if (n == "group")
            {
                sb.Append("objects:[");
                bool first = true;

                foreach(var obj in objects)
                {
                    if (!first) sb.Append(",");
                    first = false;

                    sb.Append(obj.ToString());
                }
                sb.Append("]");
            }
            else
            {
                sb.AppendFormat("{0}:[{1},{2},{3}]", "p", p[0], p[1], p[2]);

                if (r != null)
                {
                    sb.Append(",");
                    sb.AppendFormat("{0}:[{1},{2},{3}]", "r", r[0], r[1], r[2]);
                }

                if (s != null)
                {
                    sb.Append(",");
                    sb.AppendFormat("{0}:[{1},{2},{3}]", "s", s[0], s[1], s[2]);
                }

                if (c != null)
                {
                    sb.Append(",");
                    sb.AppendFormat("{0}:[{1},{2},{3}]", "c", c[0], c[1], c[2]);
                }
                sb.Append("}");
            }

            sb.Append("}");

            return sb.ToString();
        }
    }

    public class XDoc : IXHasChildren
    {
        public XRespawn respawn { get; set; }
        public double oceanlevel { get; set; }
        public string weather { get; set; }

        public List<XObject>  objects { get; set; }
    }

    class Program
    {
        static void Collect(ref List<XCollectedObject> list, IXHasChildren container)
        {
            foreach (var obj in container.objects)
            {
                var c = new XCollectedObject()
                {
                    isgroup = obj.n == "group",
                    obj = obj,
                    container = container,
                    hash = obj.ToString()
                };

                list.Add(c);

                if (c.isgroup)
                {
                    Collect(ref list, c.obj);
                }
            }
        }

        static int RemoveDuplicates(ref List<XCollectedObject> list)
        {
            List<XCollectedObject> unique = new List<XCollectedObject>();

            foreach (var objA in list)
            {
                bool duplicate = false;

                foreach (var objB in unique)
                {
                    if (objA.hash == objB.hash)
                    {
                        // remove object from parent container
                        objA.container.objects.Remove(objA.obj);

                        duplicate = true;
                        break;
                    }
                }

                if (!duplicate) unique.Add(objA);
            }

            // find empty containers
            foreach (var objA in list)
            {
                if (objA.isgroup)
                {
                    if (objA.obj.objects.Count == 0)
                    {
                        objA.container.objects.Remove(objA.obj);
                    }
                }
            }

            return list.Count - unique.Count;
        }

        static int RemoveDuplicates(string json, out string cleanJson)
        {
            int counter = 0;
            XDoc doc = JsonConvert.DeserializeObject<XDoc>(json);
            List<XCollectedObject> allObjects = new List<XCollectedObject>();

            Collect(ref allObjects, doc);

            counter = RemoveDuplicates(ref allObjects);

            cleanJson = JsonConvert.SerializeObject(
                doc,
                Formatting.None,
                new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }
            );
            
            return counter;
        }

        static void Main(string[] args)
        {
            int totalCounter = 0;
            int counter = 0;

            if (args.Length == 0)
            {
                Console.WriteLine("Please specify a filename :)");
                return;
            }

            string fileName = args[0];
            string json = File.ReadAllText(fileName);
            string cleanJson = null;

            // max 10 loops
            for (int i = 0; i < 10; i++)
            {
                counter = RemoveDuplicates(json, out cleanJson);
                totalCounter += counter;

                if (counter > 0)
                {
                    json = cleanJson;
                }
                else
                {
                    break;
                }
            }

            Console.WriteLine("{0} duplicate objects found.", totalCounter);

            File.WriteAllText(fileName.Substring(0, fileName.Length - 6) + "_xxx_clean_xxx.world", cleanJson);
        }
    }
}

 

ChloeCleanTheWorld1_2.zip

  • Like 2
Link to comment
Share on other sites

  • Modz Gold Member
2 hours ago, Ayon said:

Hi Chloe

Did you want to add this Modz with the Installer to the Files section of the forums?

 

Hi Ayon, I will create an offline installer later so it can be addedto the Files section, ty.

  • Like 3
Link to comment
Share on other sites

  • 2 weeks later...
  • Modz Gold Member

Nicely done Chloe!!!    In ZEN I have 100's of panels copy/pasted and found 30 dupes.  Thank you!!!!

 

I wonder if increasing tolerance might make other things happen too.  Thank you for sharing the code as well. <3

 

~ice

  • Like 3
Link to comment
Share on other sites

  • Modz Gold Member

Offline version does not really make sence... the installer from web + auto update works good I think.

 

New version checks the file on load and give some stats, like number of unique objects and groups too.

  • Like 1
Link to comment
Share on other sites

  • 2 weeks later...
  • 2 weeks later...
  • Modz Gold Member

You're probably using a JSON file from my alpha mod. I'm writing float values instead of ints to get higher precision between editor saves/loads. The vanilla game reads them just fine, but Chloe's deserializer is probably expecting ints only 9_9

You can re-save the file with the vanilla game (loosing a couple of digits of precision) or ask Chloe to modify her code a little :)

 

  • Like 1
Link to comment
Share on other sites

  • Modz Gold Member

Well, the vanilla reads raw Json data, and Json reader seems to auto convert floats to ints when it reads into int[] arrays, so I've decided to add a little extra precision (it's probably eaten away by float math though).

  • Like 1
Link to comment
Share on other sites

  • Modz Gold Member
On 2/2/2018 at 5:54 PM, chloe said:

Okay.... yes my code expects int, if the vanilla importer has no problems with floats I will change this. 

 

The game's code is reading vector values like this:

int[] array = token.SelectToken("p").ToObject<int[]>();

With this technique the JSON serializer seems to always try to auto-convert data to the target format, so, it should work with floats unless the devs change the deserialization code. Which is pretty unlikely to happen any time soon :)

 

 

 

  • Like 1
Link to comment
Share on other sites

  • Modz Gold Member
On 2/23/2018 at 11:36 AM, Icarus said:

Hey Cloe,

 

tested the new version. But somehow it makes the files bigger. From 565kb to 651kb after running it. Strange...

 

Any Idea?

 

Ica

 

 

chloe.PNG

 

 

That is normal since the values are now stored as float and no longer as integer. I did changed that to support modded DLLs. 3dx don't care if you import floats (with decimals) or integers. So the file size is a bit bigger but after import (either with a modded DLL or the default one) you will not notice any change or different behavior but you maybe get a higher precision using a mod.

 

Kisses Chloe

 

 

@Ica: For a follow up check the conversation between Ayon and Alex on the first page.

Edited by chloe
added sentence
Link to comment
Share on other sites

  • Modz Gold Member

Thank you for sorting Chloe. Coz I am a fan of this tool, I had to test the new version straight. :D

Was asking cause people are saying ingame, that the file-size of a room is related to get lags or not, when you make a room public and it is crowded. Something like a room should not be bigger than 500kb. But I think it is more related to textures and lights in a room.

Link to comment
Share on other sites

  • Modz Gold Member
17 hours ago, Icarus said:

Thank you for sorting Chloe. Coz I am a fan of this tool, I had to test the new version straight. :D

Was asking cause people are saying ingame, that the file-size of a room is related to get lags or not, when you make a room public and it is crowded. Something like a room should not be bigger than 500kb. But I think it is more related to textures and lights in a room.

 

 

You can use it as an indicater... like the number of objects too. But there are a difference if you have just blocks or more complex objects. And sure lights and animated objects cost performance too.

 

Link to comment
Share on other sites

  • Modz Gold Member

Thank you for this really handy tool, Chloe, checked all my .world - files by it and found some duplicates - but not as many as I had feared.

As I use scaling up and down extensively, there seems not to be a but in the scaling which causes duplicates, as someone as someone had suspected here. Maybe it is not a bad idea to switch to volume scale Gizmo while trying to select objects in a group and after selection is "stable" switch to the desired Gizmo. Anyway, I was rather content with finding only between 9...30 duplicates in files of 5000...6500 objects.

 

 

Link to comment
Share on other sites

  • 3 weeks later...
  • Modz Gold Member

Chloe, 

Thank you. I'm not a coder and frankly don't have the patience to be one. But thank you for putting this out there for us builders. I'm running through my worlds cleaning them up when I came across this particular error using the latest build of your application:

Optimizing file 'C:\...\Pool Party Club.world'...

ERROR: Failed to parse file content.

Input string '0.238' is not a valid integer. Path 'objects[0].c[0]', line 1, position 264.

 

Link to comment
Share on other sites

  • Modz Gold Member
9 hours ago, 3dxBromeo said:

Chloe, 

Thank you. I'm not a coder and frankly don't have the patience to be one. But thank you for putting this out there for us builders. I'm running through my worlds cleaning them up when I came across this particular error using the latest build of your application:

Optimizing file 'C:\...\Pool Party Club.world'...

ERROR: Failed to parse file content.

Input string '0.238' is not a valid integer. Path 'objects[0].c[0]', line 1, position 264.

 

Hmm... I check this if I am home again. Currently I am in the hospital. I wonder why you have a float color value. 

Link to comment
Share on other sites

  • Modz Administrators
On 3/22/2018 at 5:03 PM, chloe said:

ERROR: Failed to parse file content.

Input string '0.238' is not a valid integer. Path 'objects[0].c[0]', line 1, position 264.

Hi Chloe

I have received the same error tonight.

I have used your Modz on this file before with no issues, but now this error comes up.

 

I was trying to think what I have added to the file that might have caused this error, but I actually think I have removed some items.

So i really do not know how I caused this error.

Link to comment
Share on other sites

  • Modz Gold Member

Just re-downloaded this, and I am still getting this error...

 

Not sure what to do here, this worked fine before latest Update.

Like this app alot :)  looking forward to using it again.

 

M.

 

 

InkedDupe Note_LI.jpg

Link to comment
Share on other sites

  • Modz Gold Member

I think that the issue is that the file format for new objects saved after 373 changed it now has a different header and the color value is also stored as a floating point value. So this program will need to be updated to work with the new file format most likely.

Edited by Niblette
Link to comment
Share on other sites

  • Modz Gold Member
On 27/01/2018 at 3:24 AM, Ayon said:

Oi Chloe

Eu corri sua ferramenta em um clube e recebi o erro abaixo? Alguma ideia sobre o que isso pode ser?

image.png

 

 

I have the same problem in a room, is there a solution? What should I do?

Link to comment
Share on other sites

  • Modz Gold Member

Hi Chloe, I've the same issue than the others (the "failed to parse file content" thing). I hope you will be able to fix that soon (I don't want to be too demanding)

Hope your health is better by the way

 

Thanks for all you do

Link to comment
Share on other sites

  • Modz Gold Member

There was an update to the save file format since patch 373. There is now a new string token {"valuetype", "float"} added to the file when saved with game version 373 and above.

Now when reading the save file when there is no "valuetype" property (the game doesn't even check for its value, I think), the file is supposed to be an old style one, with ints. When the property is set, all transform and color values are supposed to be unmodified floats (i.e. the color components should be in the 0...1 range).

Or at least something like that, I'm writing it from my head.

Link to comment
Share on other sites

  • Modz Gold Member

thank you for your answer Alex even if it was like you were speaking tagalog to me ^^'. I'm a jack of all trades but programming isn't amoung them.

 

Could you tell me how I can do to fix my duplicate object problem in my file ? By mistake, I duplicated all my work without noticing it and I continued to build... a lot... and yes I saved my work on 373 and above version... :.( 


With all due respect for Chloe's work, I would take any solution available

 

Thanks in adavance

Link to comment
Share on other sites

  • 1 month later...
  • 1 month later...
  • Modz Gold Member

Chloe  can you please help me  i keep getting an error message .  not sure if i am doing the process wrong or not.

as there isn't any real directions to run your program.

PLEASE help

i have a wonderfull build and i know there is issues this is what i keep getting.

http://prntscr.com/jzmv26

 thanks for your help.

 

Link to comment
Share on other sites

  • Modz Gold Member
On 26.6.2018 at 6:53 PM, Ivee said:

ok sorry didn't realize it wa brought up already  duh  should read first  please let us know if an updated version comes available

 

thanks..

 

 

You are using an old version. Please download and install the latest version from here.

  • Like 1
Link to comment
Share on other sites

  • Modz Gold Member

Next version will have the possibility to add a watermark to a world file to give room thieves a harder nut to crush ^^

Edited by chloe
  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


  • Google Translate

×
×
  • Create New...

Important Information

By using and viewing this site, you agree to our Terms of Use.