Criticism and Published Material

[Jim McDougall]

In the thread concerning a sword of 'the last Armenian king', there was some genuine promise in what might have been a valuable discussion on a very esoteric and most interesting topic. Unfortunately the discussion quickly denigrated into comments and criticism that quickly recalled unpleasantness that occurred several years ago which many of us will be familiar.

Manoucher is unable to post on this forum, so in my opinion it is unfair to criticize here his comments or his published material where he has no opportunity for rebuttal. I completely am in favor of Andrews decision to close that particular thread, and felt compelled to express my personal feelings concerning open criticism of the work of published authors.

Over the years there have been many instances where published works have been harshly criticized in discussions, many of them the works of earlier writers of now venerable resources (i.e. Stone). I have nothing but the highest respect and admiration for anyone who has the courage to pour thier heart and soul into a book that represents any degree of passion for the topic they study. These books, as I have always emphatically stated, even though clearly superceded by new evidence and research, served as vital benchmarks to set the course for future study. Other more recent works, such as the volume produced by Anthony Tirri received heavy criticism, though the book represents an outstanding resource for general identification of collectible weapons.

I have contacted Manoucher to congratulate him on the publication of his new book and look forward to seeing this work. He has worked hard and it would appear he has presented a very important resource on these weapons, and has proven extremely tenacious in prevailing in his scholarship in the study of them. I would consider it in the best interest of all of us as sincere students of arms and armour to focus on the strengths and positive material in the work of all authors, and where criticism or disagreement is required, it should be presented only in a constructive and courteous manner, as some of the participants here have obviously accomplished.

I would like to see these forums continue to fluorish in tandem, rather than in competition, and for us all to help each other strengthen our knowledge together.

All best regards,
Jim

[Andrew]

Jim is, as always, a true gentleman.


Folks, let's not get ourselves into situations where we carry on arguments or differences from other fora. Criticism and spirited debate are fine, just keep it civil and avoid ad hominems.

[Rivkin]

I will be honest with you - I am an idiot, so I have no idea why the thread was locked or what you are talking about right now. Genocide and other things appeared in a swordforum discussion right in the first post, and had nothing to do with our discussion here. Here it was relatively civil.

Concerning criticism of Manoucher, I did not know that he does not post here. There are certain opinions of his which I do not share, and while he always takes a persian point of view, I usually take an opposite point of view. I find a lot of things he says objectionable - he knows about it.
I also honestly believe he is a way, way greater as a scholar than I am.

I find a lot of things I said here and on swordforum _wrong_ and misinformed. Well, we all grow. It is all about personal opinions. Should we not express our personal opinions if it contradicts someone elses ? Obviously holy relics are always explosive as hell, so what. The most strange thing is that none of us is even remotely armenian catholic, so I don't understand the emotionality.

I personally believe the best thing is for moderators to clean up the messages they find offensive and reopen the thread, but obviously what's going to be is going to be. I would like to get response to the questions that were raised concerning the dating and other issues.

[Rivkin]

Ok, here is an example of peer review criticism. It is somewhat on a better side (!!!) negative review of one of my articles:

"Overall the paper is weak in style, content and relevance.... In content and relevance, the paper may have indicated interesting issues, but little is gained from this work.... not new and has been thoroughly studied...It lacks content, style and relevance. I would discourage the authors of resubmitting...."

Now this is not so bad. I skipped all the technical and paper-related commentaries, but trust me - I had received ones 1000% meaner - this is sort of business-like.

Now to my question - does it work differently when one submits to things like Met Journal ? Again, no evil will or hidden direspect, I honestly whant to know what kind of level of criticism one usually sees in the community ?

[Valjhun]

Wonderfull thoughts Jim.

I was not partecipating at that closed therad, however I have a thing to add regarding criticism.

Criticism is GREAT! why? 'cos it helps to achieve better and better solutions for the future editions. As long as it is kept on polite and fair level(and it must be) and that is the hard point. What is politeness/fairness? When politeness is a matter of education, culture and manners, fairness is what we bring in heart as individuals. Well, for example, If someone say that Mr. Authorofabadreference is an idiot is unpolite and unfair, if someone says that Mr.Authorofabadreference's book is a piece of crap is unpolite but fair, and finnaly if someone says that Mr.Authorofabadreference is an incompetent 'cos wrote such a book is polite but not fair. When we don't mention the competency of Mr.Authorofabadreference and we concentrate on his book in saying that it is definetly bad, innacurate, overpriced, and without any competence and lack of practical implementation and we support that with strong arguments? Well what is wrong with that? Criticism is GREAT! remember? Maybe some new author (or why not the same author) looking at that criticism will improve his future work. We are living in great times...

Many of us who bought a vlome for 200$ with depixeled photos, bad descriptions, lack of measures, many mistakes, bad atributions, no descriptions at all and advertised as an ultimate collectors manual for a year before it was pubilcated.... well felt a little dissapointed and why not, angry but that is not a good reason to keep ourselves below the line ofgood taste.

EDITED: I've just read the tread about the sword of the last armenian King and I'm giving full and total support to Rivkin/Ariel, even (or because ) I'm predominantly in a NihonTo business.

[wolviex]

From the Author point of view:

Writing an article or a book is in general a very difficult undertaking. Most of us want to publish something what is the most complete, most 'sourceful' and most actual on the field, and that's why decission to publish may be hard, because it means you have to decide on exact date. You know you will have only some time not entire life to finish the work. After the text is more or less complete it is sent to publisher hands and is verificated by editors who are able to make such terrible things with your text you just can't imagine. From interpratation, to changing of some of the most basis rules! For example: my girlfriend wrote article for the newest Rembrandt catalogue, with description of some of the etchings. Editor changed text without consultation with Author in places where NOTHING should be changed at all like: 'the sitting man' became 'a standing man', 'place' became 'palace' - horror. Editor didn't wanted to show "corrected" text, so it was luck my girlfriend found those errors. But it is not enough, while editor changed some overall thoughts and some of the points in the article!!! Now imagine the Author who is reading review of "his" article/book after edition! So please remember not everything is Author's fault.
Criticism is great, we agree, while it helps author to take a look on some sort of things from different point of view, to find things he didn't think of, etc..., but it is nesessary to let him defend himself during debate. Of course there are many bad and poor books because authors didn't carry it off well.

Rivkin is right: in serious, scientific magazines you can find real harsh reviews, without a pardon! Here I found always only gentleman discussion, with great distance, intuition and comradeship. Normally, we could write serious reviews pointing every error, and it shouldn't be damned by moderators, until reviewer is malicious or incompetent. As an author I always write an article or something else with preparation on criticism, if someone is not prepared for such pain after publish, shouldn't write books at all - it's just a part of author's life. No-one should feel offended if review is sensible and grounded.

Best regards!

[Ian]

I think there is a difference when one rights "professionally" and when one writes as an "amateur" author. Rivkin's example of his own experience is of the "professional" category where he submitted his work for peer review and received an unfavorable response. Professional authors expect and accept (sometimes reluctantly) the opinions of their peers.

However, the amateur or occasional author is not prepared for the unsolicited criticisms that may accompany publication. Such comments feel like personal attacks -- many may be -- and the author's defensiveness is to be expected.

Critics cannot expect the same level of scholarship from a person who is not a recognized expert in the field or an experienced writer, nor, I believe, should critics expect the same freedom to attack the work of an amateur writer as they enjoy when participating in a solicited review. Part of the role of the publisher is to edit the material from an amateur writer and have the work reviewed so that it achieves a necessary standard of professionalism. If the final work has literary flaws, factual inaccuracies, omissions, or other deficiencies, then the publisher must take some responsibility also.

The field of writing about swords, especially those outside Europe and Britain, is not populated with many recognized experts. There are some who are knowledgeable but very few who would take the time to research such a book thoroughly and produce a scholarly work. Robert Elgood is one such person, but I'm struggling to come up with other contemporary authors who have a similar standing in our field. Perhaps Christopher Spring's African Arms and Armour is of similar quality. Going back a few years, we can think of other authors who spent many years to produce lasting efforts (e.g., Rawson, Figiel, and Stone) but they are deceased and it was a case of one or two books from each of them.

There's just not a lot of top talent writing about ethnographic edged weapons. Most of what we have seen in recent years comes from enthusiastic amateurs who wish to share what they have learned. And they get a lot of criticism for their efforts. Should they not have tried to do so? I think we are better off having their efforts available for discussion.

Perhaps the problem is not so much the quality of their work but our expectations of what it should be.

Ian.

[Andrew]

This has, historically, been a place where friends could gather to casually discuss a common, esoteric passion. Have there been arguments, and heated, spirited debate? Certainly. But one of the nice things about this forum is that it is almost completely free from so much that makes similar places on the web so unpleasant. I want it to stay that way.

Were we to abandon this perspective, I predict this place would quickly degenerate and many participants would stop posting. I, myself, have stopped posting on other fora for this reason. We've all seen it.




**Please don't turn this thread into a rehash or discussion about my decision to close the Armenian King thread. If any futher comments are made about that, I will lock this thread. Thanks.**

[wolviex]

Dear Ian!
You touched very interesting and important problems, especially about experts and amateurs. I would like to notice, that weapons-knowledge-field is somehow special between other spheres of collecting. Probably nowhere else between collectors you would find so many (sorry for the word) maniacs and lovers from every field of interest - from history and art history experts to fantasy lovers (sorry to put both on opposed sides - it's not the point). Unlike anywhere else, you'll find many, many amateurs of different arms - that's why every Author should consider and should be prepared for! 70% of his readers will be amateurs not experts, and almost every one of them believes to be an expert (and some of them are) on his field. If the discussion and accusations to the author arise, Author should have a chance to response - that's the main rule, which let the author answer on any objections - or prove his right.

[fearn]

Speaking as another author (although not on swords), I'll respectfully disagree with Valjhun's opinion that criticism is great.

Criticism is simply criticism. On the bad side, it can be a cover for personal (aka ad hominem) attacks, it can be simply wrong, or it can be useless. My favorite example of the last one was a comment I received on my master's thesis draft all too often from one person: "It's wrong. Fix it." If I was lucky, he would at least point an arrow to what offended him, without saying why it did. Given that there were all sorts of possible wrong choices but only one right choice, it took months to get what he wanted. Another reviewer got through the same process by sitting down with me for 30 minutes and explaining what he saw as problems and what he thought I needed to do to fix them. That 30 minutes vs. months dichotomy stuck in my mind. But I digress...

On the positive side, criticism identifies legitimate problems, and ideally (in my opinion), it offers better alternatives. Personally, this last is my favorite, although it can be misinterpreted as egotism on the part of the critic. All too often, something might be objectionable, but there might not be a better option. A critic making the effort to come up with a better alternative can discover that, and shape his comments accordingly.

Finally, criticism can stem from an honest difference in opinion, as Rivkin noted. These are fine, although they can get tedious when they are repeated, especially for years (as in many academic battles).

Obviously, I prefer to see more useful criticism, but that's my bias. There's plenty to disagree with here.

My 0.02 cents,

F

[ariel]

Ian,
I find it difficult to agree with your thesis that only "professionals" should be subjected to criticism. Every person who expresses an opinion, either oral or, more pertinently, printed, passes it to the public domain. From there on, this opinion becomes a fair target to counterarguments and, yes, criticism. This crititism should NOT be directed at the publisher whose role is technical and whose motive is to make money, but to the author himself, who is the source of the presumably fallacious or objectionable content. Even peer-reviewed publications do not enjoy immunity from criticism, whether because of some negligence of the original reviewer or when new data become available.
None of us have seen the forthcoming book by Mr. Khorasani but many plan on buying it (I do).Until this book is out, any critique of it is inappropriate. After that, it will become a fair target and the opinions will be divided, not dissimilar to any other book. An argument that this is a work of an "amateur" that should enjoy relaxed standards of excellence does not appeal to me . I am sure that Mr. Khorasani would also find it insultingly patronizing if a fruit of his 8 year long labor is not viewed as deserving serious treatment. After all, people are still critiquing Dante and Shakespeare, very gifted amateurs by your own definition because they did not subject their works to peer review. Tirri's book was justifiably criticised by people who wanted to have an exhausive academic treatise and justifiably praised by others (myself included) who wanted to see real collector-grade weapons.
The argument about the Armenian King's sword was aimed not at Mr. Khorasani but at the obvious misattribution of this artefact by the museum that fell victim to a legend. I see no reason why the stance by the museum's owners cannot be challenged. It has nothing to do with their religious beliefs or with any revision of Armenian history. It has to do with shoddy scholarship and a lot of wishful thinking. These are legitimate reasons to re-examine the sword and if the bubble of the legend bursts, so be it. I applaud Mr. Khorasani's decision to examine this sword personally. Any final conclusions reached by him and posted on the Internet or published elsewhere will generate controversy, either by the supporters of the legend or by it's opponents. And this is how it was for generations, and this is how it should be.
But the issue is much deeper than that. This Forum's atmosphere is characterized by openness, " no-holds-barred" approach and refusal to bow to any self-appointed "authority". We do not have sacred cows, nor do we venerate cattle breeders. Of course, we all value and respect the experience of some of our members in Moro weapons, Krises, SE Asian arms, Turkish or Caucasian blades etc. I would not dream of keeping a particular opinion about a Dha if you, Andrew and Mark classified it as something different. But, equally, I would not hesitate to challenge you to back up your verdict with reasoned arguments. Such attitude was always accepted by every Forumite as a sine qua non of our small commune.
We can, and should, demand acceptable standards of discourse, but the freedom of expressing one's opinion and bringing up pertinent facts should not be curtailed.
If we enter the slippery slope of agreeing with everything and everybody, subjugating our freedom of expression to whims and fancy of a "guru" and living in fear of offending somebody's ego by challenging his pet idea, we face a real danger of becoming an insipid , mutual admiration group exchanging meaningless pleasantries. And that would be sad...
Best wishes to everybody.
Ariel

[VANDOO]

I THINK THE KEY IS CONSTRUCTIVE CRITICISM RATHER THAN PICKING OUT SOME PARTS AND USING SUCH INACCURATIES TO CONDEM THE WHOLE.
WHEN YOU CONSIDER THE DIFFICULTY INVOLVED IN WRITING A REFRENCE BOOK ESPECIALLY ON SOMETHING AS COMPLICATED AS ETHINOGRAPHIC WEAPONS YOU CAN APPRECIATE THE EFFORT THAT GOES INTO IT. FIRST YOU WILL HAVE TO USE REFRENCES FROM PREVIOUS SOURCES AND WILL BE EXPECTED TO INCLUDE THE BEST KNOWN SPECIMINS OF THE TYPE. AMONG SCHOLARS THERE ARE OFTEN CONFLICTING VIEWS ON ALL PREVIOUS WRITEN REFRENCES AND ITEMS AND BOTH SIDES WAITING TO POUNCE IF YOUR VIEW IS DIFFERENT.
OFTEN NEW BOOKS ARE JUST A REHASHING OF PREVIOUSLY WRITTEN MATERIAL WITH NICE PICTURES AND LITTLE NEW FIELD WORK. I THINK ITS GREAT IF A AUTHOR ACTUALLY GOES OUT AND RESEARCHES NEW UNDOCUMENTED COLLECTIONS AND TRYS TO RELATE THEM TO THE BEST KNOWN AND DOCUMENTED COMPARITIVE EXAMPLES. ALL REFRENCES ON ETHINOGRAPHIC WEAPONS WILL HAVE SOME MISTAKES, OFTEN DUE TO MISENTERPRTING OR GETTING BAD INFORMATION FROM THE PEOPLE YOU ASK FOR HELP EVEN IF THEY ARE FROM A TRIBE WHO USED TO USE SUCH WEAPONS. STONES GLOSSARY HAS SUCH INACCURACYS BUT IS STILL A MONUMENTAL WORK NOT JUST A MERE REFRENCE AND HAS MORE GOOD INFORMATION THAN MISTAKES.
THE SWORD MENTIONED IN THE OTHER POST HAS BEEN TREASURED AND REVERED BY THOSE WHO BELIEVE IT TO BE THE LAST KINGS SWORD. TO BORROW A KERIS TERM THAT WOULD DEFINITELY MAKE IT A PUSAKA TO THEM AND THEIR BELIEF SHOULD BE TREATED WITH RESPECT REGARDLESS OF ITS ACCURACY. THAT BEING SAID IF THEY ARE WILLING TO LET A SCHOLAR EXAMINE THE RELIC HE CAN MAKE HIS OWN CONCLUSIONS AND PUBLISH A PAPER ON HIS FINDINGS AND HIS OPINION ON THE ITEM. I PERSONALY THINK IT IS GREAT THAT SOMEONE WOULD SPEND THEIR TIME AND MONEY TO GO DO THE REASEARCH NEEDED TO GIVE A GOOD INFORMED ASSESMENT AND OPINION ON THE SWORD.

[Ian]

Originally Posted by ariel

Ian, I find it difficult to agree with your thesis that only "professionals" should be subjected to criticism. Every person who expresses an opinion, either oral or, more pertinently, printed, passes it to the public domain. From there on, this opinion becomes a fair target to counterarguments and, yes, criticism.

Ariel: I'm not suggesting that only "professionals" should be criticized, but rather that there is a level of expertise expected of them and therefore one may be more critical of the efforts of an "expert" when the effort falls short of an expected professional standard. But is it reasonable to expect everyone who publishes on a subject to be held to the highest and most stringent criticsm as if they were indeed a leading expert? These books, after all, are not scientific journals -- many are closer to works of art and history than works of science. That's not to say that art and history have no standards, but those standards are different and more subjective in nature.

This crititism should NOT be directed at the publisher whose role is technical and whose motive is to make money, but to the author himself, who is the source of the presumably fallacious or objectionable content.

A non-professional writer needs an editor, who should be held responsible for some of the style, accuracy, etc. That editor usually works for the publisher. Good publishers are concerned about accuracy and presentation. Yes, the author is ultimately responsible for the accuracy of his/her work, but editors often contribute as well and share responsibility.

Even peer-reviewed publications do not enjoy immunity from criticism, whether because of some negligence of the original reviewer or when new data become available. ...

Apples and oranges, my friend. The books we are talking about are far removed from peer-reviewed publications and perhaps too subjective to be treated in a similar manner.

Ian.

[B.I]

wow. it seems the sword has stirred up a hornets nest. in hindsight, i would have kept my mouth shut as all i contributed was an opinion and an added diversion which was not needed, nor asked for. i think that any cross-forum discussion must be avoided as there is obvious animosity and issues at hand. in an ideal world, we could flit from one to the other but it seems that by doing so you are only becoming an open target.
i think the world is a big place and enough room for different opinions to happily sit along side each other without butting heads.
i like to think that i take my studies quite seriously, and hold every book in my library quite dear to me. i will buy any book that has anything to do with my subject, whether good or bad as until you have read everything written already, it is then that you can happily discard it all and form your own opinion :-) my library hold most books that are oriental arms related, and many that step into iconography and history.
however, there is a clear difference between what i class as 'academic' and 'picture book'.
i only distinguish this classification after i buy the book, and i never critisize the importance of a book as all have some use (i love good pictures!).
if i pay $100-$200 for a book, then i should be entitled to an opinion on its contents.
if an author considers himself an expert and wants to move the world with his opinions, then he must know that he is up for both praise and rebuke. on the other hand, if a book is written in a general sense, then it should not be rebuked for being too general.
i applaud anyone that manages to publish as this alone is not easy. so, for someone to achieve this, and also to devote an amount of years to put the project together, then he should be commended for doing so.
BUT, his work will, and MUST be up for appraisal. i know many published authors and not one of them will convince me (with a straight face) that he did so for personal gratification. all want to put their opinions forward and any decent author will willingly accept critism for his work, as long as it is constructive.
i think this is the key, as vandoo rightly says. anyone who thinks tirri's book was meant to be academic should read it again. 20-40 words per item can hardly be in-depth academic. this book cant be compared to allens persian steel (for example) which has 600 pages on one specific subject. both have uses but in a different way.
elgood has openly admitted on a number of occasions that he has put forward theories in the hope that they will provoke discussions (for or against) which could further known knowledge by bringing a discussion out in the open.
there are a number of authors that spend their 'careers' devoted to the research and publication of hopefully new material (whether in book or article format). to name a few - alexander, elgood, nicholle, zaky, gorelik, miller, melikian-chirvani etc. also, museum related - stronge, buddle, zygulski (michals friend). then there is related subjects to arms (numerous authors).
none of the authors i have listed write 'general' books on arms (with some exceptions of course). each tries to push new knowledge forward and uses pre-written texts as a foundation to either start from or to argue against. none use known knowledge as an assumption (ahem, with exceptions of course). i travelled to italy to meet an author that spent his whole career searching for publishers to allow him to research a subject (chosen by them). his books catered for not just arms and he dove into subjects he knew little about beforehand, but when he finished, he managed to create theories that stood up on their own.
saying that, there is still always need for generalised books for the novice. i will happily buy these as well as i am of the opinion that if i learn one new thing from a book, then its was worth buying.
i look forward to this new book, and will happily pay the price for it. i will read it from cover to cover and i will have a strong opinion, for or against it. this book has already declared a large amount of images, so even if the text is ridiculous, then at least i will get a chance to see images of swords that are new to me. however, i really hope that i will get something from the literature as well.
the reason this forum is so used, participated on and enjoyed is because it is open to strong opinion, if justified and voiced in a polite way. thats my reason for being here anyway. if i have to sit here with my mouth shut even though i strongly feel that what is being discussed it completely wrong, then it would be hard to participate. you cannot walk on eggshells because of fragile egos (this this i mean all authors). we learn by experience and discussion, not just reading and accepting facts without question.

[wolviex]

Originally Posted by B.I

to name a few - alexander, elgood, nicholle, zaky, gorelik, miller, melikian-chirvani etc. also, museum related - stronge, buddle, zygulski (michals friend).

Dear Brian!
I would be happy if he was, but he is ca 50 years older, and the only name I would dare to use, is 'my Master'

Michal

[Jim McDougall]

Outstanding and well thought out and presented responses everyone, thank you! It really is interesting to see the perspectives illustrated here, especially by many of you who have indeed been in the position to experience various levels of criticism in works you have completed. I must admit that personally, in the limited items I have published, I have found it disappointing that I received absolutely no criticism, either bad or good concerning them. It is true that an author anxiously awaits response when his work is presented, and truly is often prepared for the worst. Typically many authors/artists etc. are thier own worst critics, and ones own humility often dictates such expectations.
I honestly had hoped for corrections or additional information or data, or observations by more experienced or informed individuals from which I could learn. Aside from those circumstances concerning formally published material, in certain instances over the years in posting on the forums I did indeed receive some very harsh criticism quite unexpectedly which offered absolutely nothing constructive. By the same token, I have always been humbled and extremely grateful for kind words often received on my efforts to present data I had researched.

I have always found scholarly chest pounding or academic arrogance tiresome, boring and completely irrelevant to useful discussion. Too often individuals overly taken with themselves mistakenly think that degrading the work of an author, regardless if professional or amateur, makes them seem more scholarly. Too much time is wasted with such labeling, and personal animosity or misguided personality problems as well as sophomoric debates are the problems I referred to in my comments.

Perhaps I am too idealistic, but I think that valuing a work for what it is should be a personal matter. Clearly criticism serves its purpose, but should be presented with courtesy and respect. Harsh or detrimental comments questioning the integrity or knowledge of the author is unnecessary and ill placed. If an error is made, it should be noted in criticism with corrected data and support, presented as material to assist readers and accent the authors established work.

As always, remember two words, courtesy and respect. Give them to others and they will be returned to you in kind.

[fearn]

That is a very thoughtful message, Jim, and much appreciated.

The one thing I'd add is that the response "looks good" with no further comments almost invariably means that the person did not read what you wrote, in my experience. While I believe in being complimentary, I prefer details, especially when I'm circulating a rough draft for feedback.

This, by the way, is for an academic context. Around here, compliments on blades should be taken at face value

F

[Jeff D]

Darn, I really tried to stay out of this but...

The way I deal with a complex problem is to break it down to its basics. Publications are supposed to add to your knowledge. The primary question you should ask yourself is 'Does this publication add anything to my current understanding?'. If the answer is yes, then the paper is a success and was worth doing. If the answer is no then it is a failure. Simple.
This of course is a very personal and subjective grading system. A "basic" paper maybe very informative to me but quite useless to the Artzis', Jims and Philips etc. The fact that some have a greater understanding and gain less from it, does not make it a worthless, laughable, piece of crap.
Like Jim, I have been trolled into too many useless arguments with people with too small of minds to get over occasional factual errors or typo's, who also can't seem to understand that publications may be ment for only a specific portion of the community. Constructive criticism and corrections are always welcome, but, some of the spiteful bile seen is completely inappropriate and quite frankly, pathetic.

My $0.02 CDN (which is getting close to $0.02 USD)
Jeff

[ariel]

Originally Posted by Jeff D

Darn, I really tried to stay out of this but...

The way I deal with a complex problem is to break it down to its basics. Publications are supposed to add to your knowledge. The primary question you should ask yourself is 'Does this publication add anything to my current understanding?'. If the answer is yes, then the paper is a success and was worth doing. If the answer is no then it is a failure. Simple.
This of course is a very personal and subjective grading system. A "basic" paper maybe very informative to me but quite useless to the Artzis', Jims and Philips etc. The fact that some have a greater understanding and gain less from it, does not make it a worthless, laughable, piece of crap.
Like Jim, I have been trolled into too many useless arguments with people with too small of minds to get over occasional factual errors or typo's, who also can't seem to understand that publications may be ment for only a specific portion of the community. Constructive criticism and corrections are always welcome, but, some of the spiteful bile seen is completely inappropriate and quite frankly, pathetic.

My $0.02 CDN (which is getting close to $0.02 USD)
Jeff

Great point! Every book is written for a specific audience, for a specific purpose and, let's not forget, for a specific group of reviewers.
Several years ago, I submitted a grant proposal to a Federal agency. The administrator assigned it to a Study Section that had no expertise in the area and my appeals to re-assign it were rejected (" We have specialists").
Well, it came back with a priority score (grade) that put it in a 96th percentile, meaning that only 4% of all submissions were as bad as mine. Having read the reviews, I saw that my worst predictions about the suitability of this particular group of reviewers came true: they had no idea what it was all about. One review consisted of a single sentence: " This is the classic chicken/egg question". Not being able to understand the topic, they did the safest thing: rejected it. Not having time to re-write the proposal, I changed a single word in the title; this forced the administrator to assign my proposal, as a new one, to the Study Section with expertise in the area. It came back with great reviews, was put in a 4th percentile (the best 4% of all proposals ever reviewed by this Study Section) and I got all the money I requested.
I made a slide of both summary statements and still show it to the fellows I lecture on "grantsmanship".
There is another, even more vile, reason for bad critique: personal animosity and envy. I apologize for bringing a specific example and hope I do not inflict unnecessary pain on the author. But this is the best illustration I know how NOT to critique.
No recent book on arms and armour was attacked as viciously and unjustifyably as "Islamic Weapons" by Anthony Tirri. This book was not intended to be a scientific treatise on the construction, development and cultural/religious elements of Oriental weapons. This is the task that belongs to museum exhibitions and collections and to professional weapon historians such as Elgood, Astvatsaturyan, Miller, LaRocca, Zygulski and many others. Tirri's book was, and is, a beautiful exhibition of collector-grade weapons. As such, it fulfilled it's intended purpose of showing ordinary collectors what weapons they may encounter in real life.
Did it have factual errors? Yes. Was the overall title appropriate for a book that included distinctly non-Islamic weapons? No. Could it be criticised on these grounds? Absolutely.
But the personal intensity, the sheer vile and the rabid animosity of some of the "reviewers" went far beyond the boundaries of good taste, fairness and objectivity. They misread the purpose of the book and judged it according to their own, externally-imposed, criteria. It is like saying that a two-story colonial house utterly failed as a convention center. Even now, some of them are still foaming at the mouth using insulting and, perhaps, libelous statements like "plagiarism", "outright ignorance of history", "shameful disgrace of a book", "unqualified author", "blatantly false information" etc. Some stoop as low as to accuse the author of publishing the book to inflate the price of his collection. This comes from the same people who insist on high academic standards and whose contribution to the field includes an essay on how to dress like a pimp.
This is not criticism; this is a true example of "'spiteful bile". And, indeed, it is pathetic....
What can we learn from all of that? First, before critiquing a book ( a sword, a house, a stew or anything else) ask yourself: do I have enough knowledge in the area? Second, what was the goal set forth by the author and can the final result be critiqued on it's own terms? Third, are there any factual errors that need to be put straight and does the correction alter the interpretation? Fourth, how much of our critique is driven by objective facts and how much personal baggage do we bring? Last, is our critique aimed at improving the field of knowledge or is it's purpose to denigrate the original author and/or settle some personal accounts?
Then sit back and decide whether we want to be decent human beings or spiteful scoundrels. The choice is ours.

[Tim Simmons]

Personally I have found many of the much heralded authors works rather wanting and often limited. Are essays in history really academic research in the field? I often feel I am led a trail to a dead end, particularly when the view on rather historic subjects is narrow and self convinced.

[VANDOO]

THE REALITY OF ALL REFRENCES IS THAT THE MORE BOOKS YOU HAVE READ AND THE MORE KNOWLEGE YOU REMEMBER THE HARDER IT IS TO FIND NEW INFORMATION IN A NEW PUBLICATION. IN ORDER TO WRITE A BOOK ON A SUBJECT YOU MUST USE INFORMATION THAT IS ALREADY KNOWN AND OFTEN PUBLISHED OR IT IS GOING TO BE A VERY SHORT BOOK . IF THE REFRENCES ARE GIVEN IN THE BOOK IT IS NOT PLAGERISIM BUT OFTEN A AUTHOR MIGHT DRAW FROM HIS OWN KNOWLEGE NOT REMEMBERING WHERE HE GOT THE INFORMATION. THAT CAN BE UNIENTIONAL PLAGERISM OR JUST LAZY NOT WANTING TO GO THRU EVERYTHING LOOKING FOR THE INFORMATION OR JUST NOT BEING ABLE TO FIND IT. SO THE MORE KNOWLEGE YOU HAVE THE HARDER IT IS TO GET NEW INFORMATION FROM A NEW PUBLICATION BUT YOU MIGHT GET TO SEE LOTS OF NEW PICTURES.

SO A FELLOW WITH LESS KNOWLEGE IS MORE LIKELY TO GET HIS MONEYS WORTH OF KNOWLEGE THAN THE WELL READ SCHOLAR THE SCHOLAR ON THE OTHER HAND MAY HAVE EVERY PUBLISHED REFRENCE ON A PARTICULAR SUBJECT AND WILL REMEMBER MOST OF THE INFORMATION HAVING READ IT MANY TIMES IN DIFFERENT BOOKS OVER THE YEARS. SO NEW PROVEN INFORMATION IS HIGHLY VALUED BY THE SCHOLAR BUT IS HARD TO PROVE OR COME BY UNTIL WE CAN PERFECT THE TIME MACHINE

[Rivkin]

Nice, well all members of academia have their favorite stories about reviewing process.

In my experience most of "impression negative" reviews usually contain the following, very often all in one set of reviews:

a. The paper is wrong. Since usually reviewer have not really read it, such review usually accomponied by some general statements like the "the method is unsound".
b. No one gives a damn about it. "not of interest to the general community", "used to be a hot subject, but in the past years...". Again there is really no reliable indicator to measure the public's interest in unpublished article, so its a safe bet for the reviewer.
c. It was already done. Again, the reviewer did not really read it, so he will either quote some unrelated article or will just refer you to "works of leading experts in the field".

Now our community is different. First of all there are very few books on our subject and even fewer of them are good. Second our science here, in my opinion, is intrinsically subjective. We usually can not use mathematics and produce "the probability of this sword to be from XVI century is 1 minuls less than 1 in a million". Most of our arguments would not be usable in court. For examples one just has to remember Oakeshott's debucle of accepted positions on origins/dating of certain swords. While we can form an accepted consensus, I doubt we can ever speak of proving something. To top all this, we don't have a truly professional, specialized journal with a strict peer review policy. Most of the articles we use are published in all kinds of historical journals; also they are not being reviewed by specialists in arms in armour, but by specialists in history. Most of us are amateurs (well I am).

The results are there for display.

[Tim Simmons]

I would like to suggest that "specialists" are not always at the cutting edge, burping up history is not research. The amateur has been ahead of specialists in many fields throughout history. The subjects that come to mind are astronomy, paleontology and even Darwin had Wallace, an amateur as an equal and in correspondence with the great man. Many writers/researchers in our area of interest seem to concentrate on the intrinsic value of the materials used and high art styles,{understandably admirable} a relatively narrow outlook rather than the form and its origin and cultural spread and regional relevance.