Warning: file_get_contents(/data/phpspider/zhask/data//catemap/5/ruby-on-rails-4/2.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Api 在ruby on rails上运行scrap命令时出错_Api_Ruby On Rails 4_Scrape_Scraper - Fatal编程技术网

Api 在ruby on rails上运行scrap命令时出错

Api 在ruby on rails上运行scrap命令时出错,api,ruby-on-rails-4,scrape,scraper,Api,Ruby On Rails 4,Scrape,Scraper,我正在尝试在一台新电脑上重新设置我的应用程序,并运行scrape来构建数据库。当我运行我的第一个耙刮板:刮,这是我得到的错误。我不知道为什么我会犯这样的错误,任何帮助都会让我大吃一惊。。干杯 Art West@ARTWESTIV ~/desktop/duckduckjeep-master $ rake scraper:scrape --trace ** Invoke scraper:scrape (first_time) ** Invoke environment (first_time) **

我正在尝试在一台新电脑上重新设置我的应用程序,并运行scrape来构建数据库。当我运行我的第一个耙刮板:刮,这是我得到的错误。我不知道为什么我会犯这样的错误,任何帮助都会让我大吃一惊。。干杯

Art West@ARTWESTIV ~/desktop/duckduckjeep-master
$ rake scraper:scrape --trace
** Invoke scraper:scrape (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute scraper:scrape
rake aborted!
NoMethodError: undefined method `value' for nil:NilClass
c:/Users/Art West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:17:in `block (3 levels) in <top (required)>'
c:/Users/Art West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:12:in `loop'
c:/Users/Art West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:12:in `block (2 levels) in <top (required)>'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/bin/rake:33:in `<top (required)>'
c:/RailsInstaller/Ruby2.1.0/bin/rake:23:in `load'
c:/RailsInstaller/Ruby2.1.0/bin/rake:23:in `<main>'
Tasks: TOP => scraper:scrape
艺术West@ARTWESTIV~/desktop/duckduckjeep master
$rake scraper:scrape--trace
**调用刮刀:刮刀(第一次)
**调用环境(第一次)
**执行环境
**执行刮刀:刮刀
雷克流产了!
NoMethodError:nil:NilClass的未定义方法“value”
c:/Users/Art-West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:17:in'block(3层)in'
c:/Users/Art-West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:12:in'loop'
c:/Users/Art-West/desktop/duckduckjeep-master/lib/tasks/scraper.rake:12:in'block(2层)in'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:in'call'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:in'block in execute'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:in'each'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:in'execute'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:179:in`block in invoke_with_call_chain'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/2.1.0/monitor.rb:211:in'mon\u synchronize'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:172:in'invoke_with_call_chain'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:165:in'invoke'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:150:in'invoke_task'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in'block(2层)in top_level'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in'each'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in“顶层块”
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:115:in'run_with_threads'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:100:in“顶级”
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:78:in'block in run'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:176:在“标准异常处理”中
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:75:in'run'
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/bin/rake:33:in`'
c:/RailsInstaller/Ruby2.1.0/bin/rake:23:in'load'
c:/RailsInstaller/Ruby2.1.0/bin/rake:23:in`'
任务:顶部=>刮刀:刮刀
这是我的刮刀

namespace :scraper do
  desc "Fetch Craigslist posts from 3Taps"
  task scrape: :environment do
    require 'open-uri'
    require 'json'
    # Set API token and URL
    auth_token = "b077632d17da8857e2fa92c053115e43"
    polling_url = "http://polling.3taps.com/poll"

  # Grab data until up-to-date
    loop do

    # Specify request parameters
    params = {
      auth_token: auth_token,
      anchor: Anchor.first.value,
      source:"CRAIG",
      category_group: "VVVV",
      category: "VAUT",
      'location.country' => "USA",
      retvals: "location,external_url,heading,body,timestamp,price,images,annotations"

    }

    # Prepare API request
    uri = URI.parse(polling_url)
    uri.query = URI.encode_www_form(params)

    # Submit request
    result = JSON.parse(open(uri).read)

    # Display results to screen
    #puts result["postings"].first["annotations"]["year"]

    Anchor.first.update(value: result["anchor"])
    puts Anchor.first.value
    break if result["postings"].empty?

    # #store results in Database
    result["postings"].each do |posting|

      #ADD HARD FILTER (IN PROGRESS....)
      if posting["annotations"]["make"] == "Jeep"

        #create new post
        @post= Post.new
        @post.heading = posting["heading"]
        @post.body = posting["body"]
        @post.price = posting["price"]
        @post.neighborhood = posting["location"]["locality"]
        @post.external_url = posting["external_url"]
        @post.timestamp = posting["timestamp"]
        @post.year = posting ["annotations"]["year"] if posting ["annotations"]["year"].present? 
        @post.make = posting ["annotations"]["make"] if posting ["annotations"]["make"].present? 
        @post.model = posting ["annotations"]["model"] if posting ["annotations"]["model"].present? 
        @post.title_status = posting ["annotations"]["title_status"] if posting ["annotations"]["title_status"].present? 
        @post.transmission = posting ["annotations"]["transmission"] if posting ["annotations"]["transmission"].present? 
        @post.mileage = posting ["annotations"]["mileage"] if posting ["annotations"]["mileage"].present? 
        @post.source_account = posting ["annotations"]["source_account"] if posting ["annotations"]["source_account"].present?
        @post.phone = posting ["annotations"]["phone"] if posting ["annotations"]["phone"].present?
        @post.lat = posting["location"]["lat"]
        @post.lng = posting["location"]["long"]
        @post.zipcode = posting["location"]["zipcode"]
        #Save Post
        @post.save

        # Loop over images and save to Image database
        posting["images"].each do |image|
          @image = Image.new
          @image.url = image["full"]
          @image.post_id = @post.id 
          @image.save
        end
      end

    end
    end
  end




desc "Destroy All Posting Data"
task destroy_all_posts: :environment do
  Post.destroy_all
end

desc "Save neighborhood codes in a reference table"
task scrape_neighborhoods: :environment do
  require 'open-uri'
  require 'json'

      # Set API token and URL
      auth_token = "b077632d17da8857e2fa92c053115e43"
      location_url = "http://reference.3taps.com/locations"

    # Specify request parameters
    params = {
      auth_token: auth_token,
      level: "locality",
      country: "USA"
    }


# Prepare API request
uri = URI.parse(location_url)
uri.query = URI.encode_www_form(params)

    # Submit request
    result = JSON.parse(open(uri).read)

    # Display results to screen
    # puts JSON.pretty_generate result

    # Store results in database
    result["locations"].each do |location|
      @location = Location.new
      @location.code = location["code"]
      @location.name = location["short_name"]
      @location.save
    end
  end




  desc "Discard old data"
  task discard_old_data: :environment do
    Post.all.each do |post|
      if post.created_at < 72.hours.ago
        post.destoy
      end
    end
  end
end
namespace:do
描述“从3Taps获取Craigslist帖子”
任务刮取::环境做什么
需要“打开uri”
需要“json”
#设置API令牌和URL
auth_token=“b077632d17da8857e2fa92c053115e43”
轮询\u url=”http://polling.3taps.com/poll"
#抓取数据直到最新
环道
#指定请求参数
参数={
认证令牌:认证令牌,
主播:anchor.first.value,
资料来源:“克雷格”,
类别组:“VVV”,
类别:“维特”,
'地点.国家'=>“美国”,
检索:“位置、外部url、标题、正文、时间戳、价格、图像、注释”
}
#准备API请求
uri=uri.parse(轮询url)
uri.query=uri.encode\u www\u表单(参数)
#提交请求
result=JSON.parse(打开(uri).read)
#在屏幕上显示结果
#将结果[“过账”]。第一个[“注释”][“年份”]
Anchor.first.update(值:result[“Anchor”])
把锚放在第一位
如果结果[“过账”]为空,则中断。是否为空?
##将结果存储在数据库中
结果[“过账”]。每次都要过账|
#添加硬过滤器(正在进行…)
如果发布[“注释”][“制作”]=“吉普”
#创建新帖子
@post=post.new
@post.heading=过帐[“heading”]
@post.body=过帐[“body”]
@post.price=过帐[“价格”]
@post.neighbor=发布[“位置”][“位置”]
@post.external\u url=发布[“external\u url”]
@post.timestamp=过帐[“时间戳”]
@post.year=过账[“批注”][“年”]如果过账[“批注”][“年”]。是否存在?
@post.make=发布[“批注”][“制作”]如果发布[“批注”][“制作”]。是否存在?
@post.model=发布[“注释”][“模型”],如果发布[“注释”][“模型”]。是否存在?
@post.title_status=发布[“批注”][“title_status”]如果发布[“批注”][“title_status”]。是否存在?
@post.transmission=发布[“注释”][“传输”]如果发布[“注释”][“传输”]。是否存在?
@post.MILENGE=发布[“注释”][“里程”]如果发布[“注释”][“里程”]。是否存在?
@post.source\u account=过账[“批注”][“source\u account”]如果过账[“批注”][“source\u account”]。是否存在?
@post.phone=发布[“注释”][“电话”]如果发布[“注释”][“电话”]。是否存在?
@post.lat=发布[“位置”][“lat”]
@post.lng=张贴[“位置”][“长”]
@post.zipcode=发布[“位置”][“zipcode”]
#邮政储蓄
@邮政储蓄
#在图像上循环并保存到图像数据库
发布[“图像”]。每个都有图像|
@image=image.new
@image.url=image[“full”]
@image.post_id=@post.id
@image.save
结束
结束
结束
结束
结束
说明“销毁所有过帐数据”
任务销毁所有帖子::环境执行
消灭一切
结束
desc“在参考表中保存邻域代码”
任务刮擦社区::环境做什么
需要“打开uri”
需要“json”
#设置API令牌和
NoMethodError: undefined method `value' for nil:NilClass
# Specify request parameters
params = {
  auth_token: auth_token,
  anchor: Anchor.first.value, # this is the line that's creating problem
  source:"CRAIG",
  category_group: "VVVV",
  category: "VAUT",
  'location.country' => "USA",
  retvals: "location,external_url,heading,body,timestamp,price,images,annotations"

}
  anchor: Anchor.first.try(:value)