Post Reply 
 
Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
gallery has been disabled
07-25-2014, 06:37 AM
Post: #1
gallery has been disabled
the gallery was eating up too many resources so i've been forced to close it off until the gallery's database is optimized. if you can help me with bugfixes and optimizations then let me know, but if you seriously don't know your stuff then don't bother. i need an expert that will do things properly and will stick to their word when they say they're going to do something. for the time being i will have to work on a fix by myself and that's going to be quite time consuming given my poor health and low experience.
Find all posts by this user
Quote this message in a reply
07-25-2014, 03:30 PM
Post: #2
RE: gallery has been disabled
What can i do to help?
Find all posts by this user
Quote this message in a reply
07-25-2014, 08:24 PM (This post was last modified: 07-25-2014 08:24 PM by chaosscizzors.)
Post: #3
RE: gallery has been disabled
Quote:What can i do to help?
unless you're a php and sql wizard, nothing.
if everything goes according to plan the gallery should be functioning within a week, but i highly doubt it will. nothing ever seems to go according to plan around here.
i guess you could... pray? if we all come together we can complete the spirit bomb and defeat gygas.
Find all posts by this user
Quote this message in a reply
07-26-2014, 05:40 PM
Post: #4
RE: gallery has been disabled
The problem I assume is that there are people who have Crawlers running and are siphoning all the resources, did you get any error messages about the gallery so we could start finding out what is the cause of it?

İmage
Visit this user's website Find all posts by this user
Quote this message in a reply
07-26-2014, 09:38 PM
Post: #5
RE: gallery has been disabled
Hope it goes back up soon but hey I can wait for those lolis Tongue

If you want to hear my singing click this link to my vocal cover thread Smile
http://www.allthefallen.org/showthread.php?tid=5199

If you wish to roleplay send me a personal message. Note that I have no limits for roleplay so you can request anything!

"We will be one... Then we shall eradicate all who are not."
Find all posts by this user
Quote this message in a reply
07-27-2014, 01:05 AM (This post was last modified: 07-27-2014 01:09 AM by chaosscizzors.)
Post: #6
RE: gallery has been disabled
Quote:did you get any error messages about the gallery so we could start finding out what is the cause of it?
i can't really show you the report. really what this boils down to is 1: thousands of people use the gallery each day and 2: we're not on a dedicated server. we just don't have the processing power to do this hundreds of times every half hour or so (guessing here).
the list of current requests in the report is several hundred long and mostly contains image fetching requests. once i set up cloudflare all those fetch requests will be directed at the cloudlflare servers so that might be enough to get things back up and running, but i still want to do some routine tasks like defragmenting the database before it goes back online.
note that the 500 error is still popping up without the gallery being open. the forum gets a lot of use too so this isn't ALL because of the gallery. the gallery is just the main offender. the forum has plenty of static content that could be pointed at cloudflare servers as well.
Find all posts by this user
Quote this message in a reply
07-27-2014, 09:49 AM (This post was last modified: 07-27-2014 09:50 AM by Malamasala.)
Post: #7
RE: gallery has been disabled
So you are heading to cloudflare? I guess it may work. I had some issues at lolibooru I think where they used cloudflare. If my search returned a lot of results, I could get "failed to connect to server" on higher page numbers. Poor SQL/PHP or cloudflare being bad, I have no idea. But it seemed to always work for page 1,2 etc. until higher numbers so maybe cloudflare can not be blamed.

I'm no expert at database speed optimization, but I can see that there is room to optimize space usage. Your database seem to be a single table. I would suggest turning it into the BCNF standard if you want to save space. (I have no personal experience in how much work it is to change an existing database, since I've always designed it before using). If you were to change the structure, you'd naturally also have to change your query and that might save you some resources in avoiding the regex-like string comparison.

Assuming you got a pretty normal table right now, you might be able to ditch the "ORDER BY id DESC" because chances are your table is already in order and the select will be using that order. (I'd say try it, and if I'm wrong I'll feel silly and if I'm right you may have boosted speed)

I don't think defragmentation will do much, but if it won't take you too much time I guess it will be worth it. (This is based on the low chance that the OS would actually decide to split a 1-5 MB picture into multiple parts of the disc. Happens more often with 100+ MB files.)
Find all posts by this user
Quote this message in a reply
08-01-2014, 12:48 PM (This post was last modified: 08-01-2014 12:50 PM by Dark_Matter666.)
Post: #8
RE: gallery has been disabled
I'm not sure if the switch to cloudflare is going to fix this issue, but I'm thinking it wont help allot, because it isn't particularly a bandwidth issue. It's a serving (processing) issue.
We are using a "shared server" currently, with 4 people to a server. This means, we are allocated about a quarter of that server's resources. If the server management notices you are using much more processing power than your fair share, then they will put limitations on your account. In this case, it was the database processing requests for the gallery which was chewing up the processing power.

As it is now, we either need to upgrade to a Virtual server, or a dedicated server to run it smoothly, otherwise, a proper overhaul of the database could definitely fix the issue, because apparently (according to the server management's report at our host service), the database is "incredibly inefficient".

So like CJ said, we are gonna need someone who really knows what they are doing, to take a proper look at the databse, because right now, a dedi server is kinda out of the question for me. Can't afford a server right now, even with the occasional donation here and there that we are getting.
Especially with my own website, where I have to buy the games I am putting up on there. Donation money goes straight into getting more content, and a little to pay for the hosting services.

[Image: bannershare.jpg]
Visit this user's website Find all posts by this user
Quote this message in a reply
08-01-2014, 12:57 PM
Post: #9
RE: gallery has been disabled
well in time something will be figured out

fuck & peace
Find all posts by this user
Quote this message in a reply
08-02-2014, 03:01 AM (This post was last modified: 08-02-2014 03:29 AM by Malamasala.)
Post: #10
RE: gallery has been disabled
(08-01-2014 12:48 PM)Dark_Matter666 Wrote:  a proper overhaul of the database could definitely fix the issue, because apparently (according to the server management's report at our host service), the database is "incredibly inefficient".

Since databases are kind of interesting, I looked up if there was any good information about database efficiency.

http://use-the-index-luke.com/sql/preface

It would seem like even though your database is "incredibly inefficient", it does not have too much to do with the database layout itself but rather about smart indexing and the way you send your queries. (But I guess for a good indexing you need a good layout)

I believe you would want something similar to this:

Code:
Table: Tags
Columns: TagId (Primary key, indexed), TagName
Example:
1, brown
2, loli
3, best

Table: ImageTags
Columns: TagRefId, ImageRefId
Both columns should together form a key, and be indexed (on TagId or together?)
Example:
1,1
2,1
2,2
2,3
3,1

Table: ImageData
Columns: ImageId (Primary key, indexed), ImageName, Directory, Owner
Example:
1, 1.jpg, C:\, Benny
2, 2.jpg, C:\, Benny
3, 3.jpg, C:\, Bob

This structure would mean that you search for brown and it begins looking in our small tags table (each row is a tag). When we find the TagName brown, we pick out the id 1 and look in next table. Since it is indexed it will be very fast to find id 1 again and the matching image id 1. Next we move to the ImageData table and again thanks to indexing we'll quickly find id 1 and return that it is 1.jpg.

The query would however look a bit more advanced.

Code:
SELECT ImageName, Directory, TagName FROM  ImageData, ImageTags, Tags WHERE TagName == 'brown' AND Tags.TagId == ImageTags.TagRefId AND ImageData.ImageId == ImageTags.ImageRefId

Alternatively I think the following works

SELECT ImageName, Directory, TagName FROM ImageData,
(SELECT ImageRefId
FROM ImageTags, (SELECT TagId FROM Tags WHERE Tags.TagName == 'brown') as CurrentIds
WHERE ImageTags.TagRefId == CurrentIds.TagId) as CurrentImageIds
WHERE ImageData.ImageId == CurrentImageIds.ImageRefId

Do keep in mind that it has been a long time since I wrote SQL queries, so I might have gotten something wrong. But the gist of it should probably be correct.

Also I skipped score and rating to keep the example clean on just searching for images based on tags. You could most likely include score and rating (if for searching) in the same manner as the Tags table and have a ImageScore or ImageRating link table. However if it is not for searching, feel free to just straight include it in the ImageData table since the splitting up is just to increase search speed.

Continued in next post.

I wanted to add another post just to point out that hopefully the php pages themselves are written in a smart way to keep queries and processing down. For example, if the query is actually re-submitted each time you click "next page", then that means you get all the data you wanted, and you just throw it away to re-query for it and use a few bits on the next page.

While your search is done, there is no need to perform a new search unless the search strings are changed. You should already have all the data from the query and all you need to do is extract some of it to render the php page.


I have a strong suspicion that currently it DOES re-query the server since I've noticed that even if a new search is not performed, going to the previous page for example may show you different images due to database having been updated with new objects. (On the other hand, I do think it is amazing that if it re-queries, it always goes very fast)

So changing this would for example cut down the query amount on a 5 page result from 5 queries to 1 query, which is 80% less server pressure. But a 100 page return query would cut down the query by 99 times.


So if I am right, and your actual problem is server pressure, I think updating how your PHP code handles the queries can save way more server resources for way less effort than changing the database.

Here is how I'd do it (not real code, just pseudo):
Code:
If OldSearchWords != CurrentSearchWords OR sqlResult == Null
   sqlResult = PerformQuery()

OldSearchWords = CurrentSearchWords

Pages = ResultSize / ImagesPerPage

GeneratePage(CurrentPage)

NextButton => CurrentPage + 1
PreviousButton => CurrentPage - 1
SearchBox => set new CurrentSearchWords

As always, it was ages since I used PHP so I don't remember all the useful things. Perhaps there is some kind of persistent variable that can be used to keep some data without having to send it each time the PHP page refreshes. (Or maybe you can use cookies)

I wouldn't mind helping out, but I suspect there is a chance I get too busy with something else since it tends to happen with most things I attempt to do.
Find all posts by this user
Quote this message in a reply
08-02-2014, 07:26 AM (This post was last modified: 08-02-2014 07:26 AM by k1u.)
Post: #11
RE: gallery has been disabled
Hi,

I've chatted with you in IRC in the past about providing resources; I would be happy to host the database for you remotely if content can stay on this server. I'm in irc.
Find all posts by this user
Quote this message in a reply
08-05-2014, 12:11 AM
Post: #12
RE: gallery has been disabled
How long untill the gallery is back up?
Find all posts by this user
Quote this message in a reply
08-05-2014, 12:45 AM
Post: #13
RE: gallery has been disabled
While it is disappointing that the gallery is down, I am perfectly fine with you guys taking as long as you want to fix it. I'm a member of ATF for the outstandingly awesome supremly masterful, community. Not the loli. Thank you for making this safe haven for my people. I love you all for your tireless hard work and dedication. You have performed a great service to a. Widely hated and misunderstood group. And you are saints for maintaining it.

http://www.youtube.com/watch?v=UhBo7X68-l0

That's how you break down an unassailably loveable character, just make him a pedophile~ Danny Sexbang 2013
Find all posts by this user
Quote this message in a reply
08-05-2014, 10:15 PM
Post: #14
RE: gallery has been disabled
Awwww :|
Find all posts by this user
Quote this message in a reply
08-06-2014, 04:31 PM
Post: #15
RE: gallery has been disabled
Does anybody know when the gallery will be back up
Find all posts by this user
Quote this message in a reply
08-07-2014, 07:38 AM
Post: #16
RE: gallery has been disabled
when ever it gets fixed itll be back up

fuck & peace
Find all posts by this user
Quote this message in a reply
08-10-2014, 02:04 AM
Post: #17
RE: gallery has been disabled
....the gallery has been disabled!?!?
https://www.youtube.com/watch?v=WWaLxFIVX1s

All bras must be removed and all panties shall fall before me!
Find all posts by this user
Quote this message in a reply
08-10-2014, 06:47 AM
Post: #18
RE: gallery has been disabled
Yes for now. They are having issues. Just be patient it will come back as soon as they can get it back up.
Find all posts by this user
Quote this message in a reply
08-10-2014, 07:16 AM
Post: #19
RE: gallery has been disabled
(07-25-2014 08:24 PM)chaosscizzors Wrote:  
Quote:What can i do to help?
unless you're a php and sql wizard, nothing.
if everything goes according to plan the gallery should be functioning within a week, but i highly doubt it will. nothing ever seems to go according to plan around here.
i guess you could... pray? if we all come together we can complete the spirit bomb and defeat gygas.

2 weeks later
Would it really helped if I pray
But more seriously is there a way to donate to the cause or something that we could do?

♪♪ I walk up to the club and I'm like, I have social anxiety and I'd rather be home right now! ♪♪
Visit this user's website Find all posts by this user
Quote this message in a reply
08-10-2014, 09:23 AM
Post: #20
RE: gallery has been disabled
find some one who knows how to fix it

fuck & peace
Find all posts by this user
Quote this message in a reply
08-10-2014, 11:55 AM
Post: #21
RE: gallery has been disabled
Any updates on the gallery ?
Find all posts by this user
Quote this message in a reply
08-10-2014, 02:59 PM
Post: #22
RE: gallery has been disabled
none so far

fuck & peace
Find all posts by this user
Quote this message in a reply
08-13-2014, 05:48 AM
Post: #23
RE: gallery has been disabled
They know someone who can fix it, that someone just happens to be unreliable due to life being annoying. I was originally going to fix the thing, but then decided to rework it from the ground up, then got sidetracked with life. The main problem with the php is they leave a lot unchecked, so there stuff that's just being dumped in error and warning logs. The SQL relies a lot on joins, which are the worst thing ever (I prefer nested queries). The SQL is a bit of a pain to optimize because it's handled by php classes and functions they made themselves and it's just not a clear design. I had the files for the gallery floating around somewhere, but I changed a lot of stuff on my computers, so I have no idea where the files are or if I even have them.

My suggestion would be to setup a github account and set up the gallery software on it as a group project. Then anyone in the community with some time on their hands and knows what they're doing can help out. Don't necessarily need them to edit code, can just be as simple as leaving a comment on a problem they noticed they might not be able to fix. I browse github on my phone every now and again, so I'd probably be able to give more reliable help.

If you have any ideas for changes in the site appearance, additions to the site, etc, feel free to contact me.
Find all posts by this user
Quote this message in a reply
08-13-2014, 01:54 PM
Post: #24
RE: gallery has been disabled
Well not to be 100% rude but i totally called this. Came back to see if the site was still around, luckily you guys managed to keep the forums online at least. I am still willing to help and get things rolling/done but the main issue is that you have to let me. Last time between people not taking anything remotely seriously and nothing getting done all the work that could have been done never got done. I have servers i can donate for the gallery to spread the load and i have the time needed to do it but i cant do it if i have to report every command line option i enter then explain what it is and then wait for the decision on if they like that idea or not to someone who doesn't actually understand IT in any regard. I even now have the ability to boast my CompTIA A+ Network+ Linux+ and Security+ certs. Working on the Cisco certs now. So if you want help and are actually willing to accept the help this time lemmy know ill try to login every now and then to check for a response. If not then, well good luck i suppose.
Find all posts by this user
Quote this message in a reply
08-14-2014, 08:18 AM
Post: #25
RE: gallery has been disabled
(08-13-2014 05:48 AM)Lambdawolf46 Wrote:  The SQL relies a lot on joins, which are the worst thing ever (I prefer nested queries).

I'm sorry but I'm slightly confused. Some examples were posted earlier, and they didn't seem to involve joins at all (and why should it, it looked like the SQL was completely flat and only contained one table?)

Still, I think the only suitable solution is to redo it properly if it is a mess. Nothing good tend to come from messes being "quick fixed" into a new messes.
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: Whywouldyouchooseme, 40 Guest(s)