last day (14 days later) » 

10:29 AM
0
Q: How to get plain text in between multiple html tag using scrapy

NandI am trying to grab all text from multiple tag from a given URL using scrapy .I am new to scrapy. I don't have much idea how to achieve this.Learning through examples and people experience on stackoverflow. Here is list of tags that i am targeting. <div class="TabsMenu fl coloropa2 fontreg"><p>...

 
Can you provide a real sample? your <p> in your question has only 1 text node so it won't show the problem you have with '//body//p//text()' and you are not showing HTML with <a>'s inside <li>'s
 
@paultrmbrth Thank u paul. I have edit question, pasted actual code.Right now I am getting text but it all comes together.Is it possible to get each and every phrases in a new line and also how to put space between word?
 
Please also paste actual HTML snippet, not made up example. Things like <li>text</li> <h1 to h5>text</h> do not help for a reproduceable case
 
@paultrmbrth I have edited question and paste actual html content and expected output.Please guide me to achieve expected output.
 
There's no "tab_description" in your input.
 
10:29 AM
@paultrmbrth didn't get your question.Please elabrate
 
you're using `sel.xpath('//*[@id="tab_description"]/ul/li[descendant-or-self::text()]').extract()`
while there's no element with id "tab_description"
 
Hi Paul
 
If you have an element, say sel, you can also use sel.xpath('string(.)')
this will give you a string representation of the element, without the tags
 
i took that code while googling for collecting text under <li></li> tag
 
it's very similar to ''.join(self.xpath('.//text()')), which concatenates all text nodes inside/under that element
 
10:32 AM
i don't have much idea about data extraction . this woulde me my mistake
now how can correct this
?
 
did you read my messages above? or was it not clear?
 
yup. umean if i wld use only sel.xpath('string(.)') , iwld able to get all text
?
i am really sorry . but i am not getting it
can u give me some sample code
 
if you have a parent HTML element that contains all the text you need,
and select it via some class or id
 
ok
then
 
example: response.xpath('//div[@id="someuniqueid"]')
you can chain a .xpath('string(.)') call
 
10:38 AM
i have multiple pages to collect data
 
response.xpath('//div[@id="someuniqueid"]').xpath('string(.)').extract()
 
so i cann't give specific id
 
so something else to identify from where to extract in the HTML
whatever you have (class, text content)
without your website it's too hard to give you generic answer
 
actually i am creating a translation tool that traslate a given word or phrase in local language
so i am collecting text from web
 
so you want only words? without structure?
 
10:41 AM
yes
only words and phrases
that only i want
 
but it only give data of specific id
 
try
`response.xpath('//body//*[not(self::script or self::style)]/text()').extract()
`
 
ok
how i will i get each pharse in new line
?
 
without sample input I can't tell you
 
10:45 AM
how how will i replace special character with space ?
 
what is special character for you?
 
ok let say jeevansathi.com
/r/t/u
 
you can use Python code for cleaning up strings
I believe you want to keep newlines (\n)
 
yes
 
have you tried the XPath I gave you above?
 
10:47 AM
trying
ok i will try out .. can i ping out directly ? if don't mind
on hangout or over skype ?
 
I don't do that kind of support sorry
always better to ask question on StackOverflow, get more eyes on your case
 
yup i am getting everything but all with special charachert
uploading file
ntent to m, u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'We will call you at the earliest after you submit the request', u'\n', u'\n', u'\n', u'\n', u'\n', u' ', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u' ', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'What type of query do you have?', u'\n', u'\n', u'\n', u'\n', u'\n', u'Questions or feedback regarding jeevansathi profile', u'\n', u'Query regarding jeevansathi membership plans', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u' ',
 
what about ` print("".join(response.xpath('//body//*[not(self::script or self::style)]/text()').extract() ))`?
 
now i have to do Python code for cleaning up strings?
 
you're getting a list of strings, you can join them
 
10:55 AM
ok let me try
yes it worked
ndia - Any City Metro Cities Delhi NCR Mumbai Region Kolkata Chennai Bangalore Hyderabad/Secunderabad Pune/ Chinchwad Andaman & Nicobar Islands Port Blair Andhra Pradesh Adoni Amravati Anantapur Bhimavaram Chittor Cuddapah Eluru Gudivada Guntakal Guntur Hindupur Hyderabad/Secunderabad Kakinada Karimnagar Khammam Kurnool Machilipatnam Mahbubnagar Nandyal Nellore Nizamabad Ongole Pal
but still most of cities coming in line ?
can we make all citis to new line
 
cities from jeevansathi.com ?
 
like
yes
and it also consist lots of blank space . can we remove that ?
 
you'll have to work on this yourself. I can't help you with such specific questions
paste relevant sub part of the HTMl and ask in StackOverflow if you get stuck
you'll have to loop on the cities, and not join them in that case, but consider them seperate
you maye want to try html2text too
 
sure .. thank u so much paul
u helped me a lot
:-)
keep guiding us
 

last day (14 days later) »