Veröffentlicht am 2014-10-04 02:14:0 in /kc/

/kc/ 41434: Catalog

syntetyc Avatar

May I make a catalog script, mods?

1. May I make a catalog script, mods?
2. May I use /prog/ for this?
3. What do mods let KC be?

1. About Catalog Script.
I feel KC has 3 matters which may be added. Catalog, floating post form and function of showing 'You' in the reply which was made to me. I already made a script for 'floating post form', so now I'll try to make a script for generating a catalog. I mean catalog like those in 4chan or other most of all chans have.
However, lacking of catalog or OP search function is also a part of characteristic of KC anymore. It force us read or see many of thread than other chans. Therefore, people who don't want to read or see other thread will leave here. People who don't want to read longer sentences will also leave here, because why they hate longer sentences comes from their poor reading skill, which consumes more time than well-educated expert. Repeatedly, KC forces us to read more sentences to find or join a thread, this make a pressure to remove people who don't have enough reading skill.
But lacking catalog function scatters residents. We can't find the thread we want to participate easily. Sometimes we made a multiple thread just because we couldn't find existence of similar thread. And personally I think time for searching thread is waste of time. I want to consume time for considering or making a reply, not for searching.
So, I want to make a catalog. May I make it although I know the catalog have both good and bad faces.

2. May I use /prog/ for this project? I want to separate technical matters and administrative discussion. I'm a beginner of Javascript, so any advices are welcome.

3. Still I wonder what mods let KC be. Now I don't have an idea because I haven't seen many threads so far because of lacking catalog. This is one reason why I want a catalog.
Anyway system character and criteria of moderation are key of chans' character. And no catalog is big. If mods don't want to make a catalog, I shouldn't make it.
If you want, I can add a tag for search. For example, pre-defined keywords for search will become a tag if people want to use it for keyword, like #news and #IWO. If the script download the keywords from KC, you can control it at any time.

I'll add graph of post/thread ratio. You can see KC's ratio is low, because people are scattered.
(Number of new threads are not acquired well in some pic.)

marrimo Avatar

KC/int/ and KC/b/.
You can see average posts in a thread is about 20.
(The graph contains the down time of yesterday.)

wahidanggara Avatar

4chan /int/, /a/, /b/ and /jp/.
Average posts/thread is about 60-80.
(Number of thread is not acquired well in graph of /b/.)

andrewgurylev Avatar
andrewgurylev:#41437 and 8chan/v/.
(Number of threads is not acquired well in 8chan doesn't contain enough points.)

necodymiconer Avatar

A client sided catalog is a bad idea, because every user had to scrape all 20 pages on every refresh, which could lead to a high load on the server.

There is an experimental version of a website based catalog with regular refresh intervals and a basic search function.

vicivadeline Avatar

What is the name of the script?

starburst1977 Avatar

> which could lead to a high load on the server.
No. Each page's HTML are about 35KB, so it invokes 35*20=700KB transaction. This is equivalent of a middle sized image, not so heavy.

andychipster Avatar

You are forgetting the network overhead, so a single transferred page inflates to ~160KB on the wire. Try it for yourself with tcpdump/wireshark.
That times 20 for every user every X minutes.
And then you still have to fetch the pages and parse them via js. One page takes around 2 seconds, so >30 sec for the whole set unparallelized + 10 seconds parsing via libxml2, a C library, js will be slower of course. Not exactly a fluent experiences, especially is you want to refresh the catalog often.

As stated it's not directly a (user)script, but a webpage.

sava2500 Avatar

> As stated it's not directly a (user)script, but a webpage.
> implying you are one of administrators
Why don't you release it, or make an official(server-side) catalog? That is the best solution technically.

As for speed of Javascript, it's not a matter, because I'll display it progressively.

And I checked consistency of transferred size between WireShark and developer's tool in Chrome. They are consistent.
Now I'll download only required things, because I'll use XMLHttpRequest(), which don't interpret HTML. Pic.1 shows that, I got 0-11 pages, 35-50KB*11pages, they aren't inflated, that's all.

If you use iflame to load, it interprets HTML and loads all particles in the HTML, that inflates transaction. I'll show that in pictures about load, reload, no-cache(load), no-cache(sorted by size). If you load it at the first time, it will be similar with no-cache. You can see it downloads heavy garbage of banner or javascripts. Of course it loads pngs, but this is required when server-side catalog are released, so matter of discussion must be focused on garbage. And repeatedly, XMLHttpRequest() don't load these garbage, don't inflate its size. When the catalog shows it, they may require javascripts, but that was cached and no transaction will be occurred.

vaughanmoffitt Avatar

Pics of load, reload, no-cache and no-cache(sorted by size).

johnriordan Avatar

> Why don't you release it
It thought I already linked to it, my bad:

>or make an official(server-side) catalog?
To the author it was more of a little hack to play with a programming language, than a serious project. But the complete source code is available, so anyone with a http server can set it up.

kuldarkalvik Avatar

Thanks, I'll try it.
BTW, you know what technique the site uses, this means you are the man or near the man who wrote this script, right? Then, may I ask questions for you if I encounter problems?

Anyway, thanks.

degandhi024 Avatar

Do you know the refresh rate of the site?

heikopaiko Avatar

Sure, there is also an email address listed on the page.

15 minutes for /b/ and 30 minutes for /int/.

rahmeen Avatar

> 15 minutes for /b/ and 30 minutes for /int/.
Why? 5 minutes for /int/, please.

> Sure, there is also an email address listed on the page.
Sorry, I can't find it, but now I can contact with you here, this is OK.

tjisousa Avatar

>Kontakt: [email protected]

> Why? 5 minutes for /int/, please.
I'll look what i can do (it's 15 minutes now), but an official response from the administration on that whole topic would be nice.

franciscoamk Avatar

Now I made a prototype of catalog.
If you want to see, I'll upload it.

So far, it is quite light, because I implement a software-cache in the script. But often it crashes when I drag the window to move with catalog shown.

nasirwd Avatar

I uploaded it since previous version has a middle level bag.
If you are interested, read >>/jp/36393 and use it, but it contains other problems.

ryanmclaughlin Avatar

Give those guys a goddamn catalog already or at least an API.

cboller1 Avatar

Sorry, I gave a workaround for stability issue. Try it, please.

joshhemsley Avatar

We will add an official catalog this year, please don't build anything that pointlessly increases server load, because having an arms race trying to get rid of it would be a waste of everyone's time. Thank you.

kazukichi_0914 Avatar

Sorry,I'll never stop developing it until you release it. And I'll implement cross-board(working now) and cross-site catalog, so it have a point, which will remove the barrier between sites.
And you can see, my script is quite light at least in client side. As for the server load, I mentioned above, and I checked it's going well. My script doesn't load any wastes(ex. extra javascript), just load pictures which I really need. Comparing to the official catalog in server side, client side catalog needs some extra HTML, this is inevitable. In my estimation, it's about 700KB-1MB. You are in the advantageous side in this aspect. You can prepare lighter, swifter catalog than me technically. If you prepare good one, residents will use it instead of heavy script in client side which I wrote. You said you'll introduce official catalog soon, this means it's a matter of technical, not administrative or political. So it's quite simple. You should give us good one, then we'll use it, that's all.

sreejithexp Avatar

>Sorry,I'll never stop developing it until you release it.

Good to know who I'm dealing with then.

>As for the server load, I mentioned above, and I checked it's going well.

It's beyond your means to check, really. Anyway, expect me to break it or block people who I suspect are using it if it seems to be causing problems. Thanks for your cooperation.

devankoshal Avatar

By the way, the point is not downloading of images or Javascript, that is not a problem at all. It's that you might be downloading a lot of pages that are normally not cached and will therefore have to be generated on the fly. And that might become a problem.

vladarbatov Avatar

> It's that you might be downloading a lot of pages that are normally not cached and will therefore have to be generated on the fly.
Then, cache it please. Just 20 pages, 1MB. Lower pages allow us to use longer lifetime. About this point, I felt you should adopt longer lifetime at least for the banner. I saw 200 instead of 304 many times just because of revised banner.
And you said the bottle neck is not network, but CPU usage, that is beyond me, that is really your business.

iamgarth Avatar

Dude, if that's your position and you have already declared that you more or less don't give a fuck even after I have asked you nicely not to do anything that might put extra load on the site, can you please give me a reason not to ban your silly threads where you advertise this thing outright?

Because if you can't, that's exactly what I'm going to do.

stushona Avatar

> Then, cache it please.
As far as I know that would entail either changes to the caching model or writing 21 pages of html per post instead of just two.

kriegs Avatar

Please just provide some APIs for developers. Like a cached RSS-Feed with all Threads, Metadata about the images (md5 hash) in html and feed, etc.
This would improve userscripts while reducing the load caused by these scripts.

ionuss Avatar

I guess it wouldn't hurt to provide a longer RSS feed (right now it's just 20 threads on /int/).